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Abstract 

lJ We analyze the state of the art of content-based retrieval in Earth observation image archives 

^^ focusing on complete systems showing promise for operational implementation. The different 

^ paradigms at the basis of the main system families are introduced. The approaches taken are 



considered, focusing in particular on the phases after primitive feature extraction. The solutions 
envisaged for the issues related to feature simplification and synthesis, indexing, semantic labeling 

^> are reviewed. The methodologies for query specification and execution are evaluated. Conclusions 

l-H 

are drawn on the state of published research in Earth observation (EO) mining. 
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1. Introduction 



Earth observation (EO) archive volumes are slowly approaching the zettabyte scale. The 



C^ assets they contain are largely under-exploited: the majority of records have never been accessed. 

o 

CsJ The situation is exacerbated by the growing interest in and availability of metric and submetric 

L" resolution sensors, due to the ever-expanding data volumes and the extreme diversity of content 

■^ 

k>< in the imaged scenes at these scales. Data from missions such as the ESA Sentinels will be 

5^ open and free, thus a much larger audience will want to use them. Interpreters to manually 

annotate archived content are expensive and tend to operate in applicative domains with stable, 

well-formalized requirements (e.g. the military) rather than on the open-ended needs of the 

remote sensing community at large or of broad efforts like GEOSS (King et al., 2011). 

Semi-automatic tools to analyze, label, summarize the contents of the archived image products 
need to be rendered a standard component of pay load ground segments. 
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Figure 1: Papers per year in references database. After a seminal period (up to 1995), the late 1990s have seen 
a significant growth of the EO mining domain in research terms. Years 2000-2005 have focused on a number of 
implementation efforts, with an explosion in the last five years of the last decade. 



Development is currently very active in this domain. Workshops like the ESA-EUSC-JRC 
IIM Workshop (http://rssportal.esa.int/tiki- index. php?page=2011_ESA-EUSC-JRC) 
are organized to discuss about the state of the art in Earth observation information retrieval. 
Special issues of Journals such as the IEEE Transactions on Geoscience and Remote Sensing 
have been dedicated to the topic (Datcu et al., 2007, 2010). Advanced, specialized problems 
related e.g. to object-based mining methodologies have been the subject of a number of contributions 
on publications such as the ISPRS Journal of Photogrammetry & Remote Sensing. The present 
work analyzes more than one hundred references with global geographical representation and 
whose temporal distribution is represented in the plot in figure 1 . 

In the US, both research oriented systems and operational tools are under active development. 
The first category includes tools like the I^KR at the Mississippi State University (King et al., 
2007) and VIS-STAMP at the University of South Carolina (Guo, 2003). The second one 
comprises VisiMine (Tusk et al., 2003), Insightful GeoBrowse (Marchisio et al., 1998), Earth 
Perspective's GeoIRIS (Shyu et al., 2006) and the RBIR system at Oak Ridge National Laboratory 
(Tobin et al., 2006). 

European activity in the domain is rich and diverse. It is embodied in systems such as ESA 



KIM (Schroder et al., 2000) (Datcu et al., 2002), KEO (Datcu et al., 2003) IIM-TS (Bovolo 
and Bruzzone, 2007), installed at ESA, DLR, CNES and MEA (Mantovani et al., 2009), in 
the Finnish prototype (Molinier et al., 2007b) based on PicSOM system developed at Aalto 
University and in SemQuery (Sheikholeslami et al., 2002), in the french PLATO (Rital et al., 
2008), the italian Centro Nazionale Multi-missione/Data Mining component at AST (Garramone, 
2009) and in the upcoming TELEIOS EO virtual observatory (http : //www . earthobservatory . 
eu/) and in the ESA/DLR EOLib connected directly to the Sentinel payload ground segments 
among others. 

The Brazilian GeoDMA (Korting et al., 2008; Pahl, 2008) is currently constrained to the 
analysis of limited datasets, yet shows promise for contexts involving larger data volumes. A 
further interesting component is TerraLib (Camara et al., 2008), an open-source GIS software 
library that handles spatio-temporal data types (events, moving objects, cell spaces, modifiable 
objects) and allows spatial, temporal and attribute queries on the database, providing functions 
for data conversion, display, exploratory spatial data analysis and spatial and non-spatial queries. 

Activity in the Asia and Pacific region is represented by innovative prototype systems such 
as those in (Ruan et al., 2006) and (Li and Bretschneider, 2007). 

In the face of all of these research and development efforts, archive owners and operators 
are attempting a rapid transition to operational systems for automatic data annotation connected 
directly to satelhte payload ground segments. Different options are available among architectures 
and paradigms for pre-operational Earth Observation digital asset management systems, data 
structures and algorithms. In this paper, we conduct a critical analysis of the state of the art in 
this domain, specifically focusing on validated approaches aiming at operational exploitation, in 
the hope of providing a contribution to this transition. 

This work is structured as follows: section 2 describes the review methodology. In section 
3, a basic abstract description of an EO mining system is used to derive a decomposition of 
its functionality into modules. The next sections analyze in detail the different choices taken 
with respect to specific processing steps: section 4 describes query specification and processing, 
section 4.5.1 the indexing approaches, and section 5 the data ingestion and autonomous analysis. 
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Section 6 reports a general discussion of the review results. 

2. Review methodology 

The assessment of the state of the art in EO data mining has involved discovering, aggregating, 
normalizing and analyzing new material to a pool of bibliographic sources in different iterations. 

The original sources considered have included the journals and the proceedings of the conferences 
and workshops mentioned in section 1. Items relevant to the query '(Earth observation OR 
remote sensing) AND (data mining OR content based retrieval)' according to the arXiv at the 
Cornell University Library (http://www.arxiv.org/), Google Scholar (http : //scholar . 
google . com/), Mendeley (http : //www.mendeley . com/) andZotero (http: //www.zotero . 
org/) have constituted primary material as well. Progressive aggregations have included material 
cited by these sources. A further significant source of material has been constituted by exploring 
publications citing the papers in the pool via the above literature search tools and Google Scholar 
in particular. 

The analysis of the characteristics of the pooled material by objective means has been an 
important methodological point of the conducted analysis. 

Inspired by Guns et al. (2011) and Giuliani et al. (2010), the metadata describing the 
considered references have been analyzed by developing a toolset to build a graph representation 
of the co-authorship relations among their authors. The representation obtained by this analysis 
process has been manipulated in the open source Gephi (Bastian et al., 2009) system (https : 
//gephi . org/) to obtain the graph map depicted in figure 2. In it, nodes correspond to single 
authors. Their sizes are proportional to the number of authored publications in the references 
pool, while edges, representing co-authoring relationships, are weighted proportionally to the 
considered joint publication number. The graph representation has then been subjected to a 
Yifan-Hu layout optimization procedure (Hu, 2005). 

The immediately apparent clustering resulting in the figure actually corresponds to the main 
EO system families represented in the considered literature: framework names in larger bold 
letters have been added to the diagram for readability. The characteristics of the main families 
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Figure 2: Graph representation of the references database for the present review paper. Nodes correspond to 
single authors, with sizes proportional to number of considered authored publications. Edges represent co-authoring 
relationships, with weights proportional to number of joint publications considered. The immediately apparent clustering 
obtained by a Yifan-Hu layout optimization procedure (Hu, 2005) corresponds to the main EO mining system families 
represented in the referenced literature. Framework names in larger letters have therefore been added for readability. 



Table 1 : References for the systems in table 2 

System Original Implementation References 

GeoBrowse Insightful Corp. Marchisio et al. (1998) 

GeoIRIS University of Missouri Shyu et al. (2007) 

I^KR Mississippi State University Durbha and King (2005) 

KEO DLR and ACS Datcu et al. (2003) 

RBIR ONRL Tobin et al. (2006) 

PicSOM Aalto University & VTT Molinier et al. (2007a) 

MEA MEEO Natali et al. (201 1) 



of systems in operation are summarized together with a description of their architectures into 
essential subsystems in tables 1 and 2. 

3. System description and decomposition 

A few assumptions can be made in order to characterize the nature of an EO mining system 
and of its operating domain (figure 3). 

1. The system runs on very large (petabyte) scale data base archives. Accessible data and 
metadata are available for the archived products. 

2. External support datasets are also often available, from pre-existing vector maps to information 
from additional sensors, and might in principle be used e.g. via data fusion methodologies. 

3. The main goal of an EO mining system is to help efficiently discover, annotate and retrieve 
specific data products in the archive based on a semi-automatic characterization of their 
contents / of the contents of the scenes they represent. 

4. The overall objective of this capability is to provide support to environmental understanding 
activities such as rapid mapping for disaster management, decision support for planning in 
large scale engineering, global climate change mapping. 

5. Additional important overall objectives are those related to the management and monitoring 
of the archive and of the whole remote sensing system. 

To achieve the final goal of EO archive-wide semi-automatic content discovery, annotation and 

retrieval, the methodologies envisioned by the EO mining systems described in the considered 

literature references vary from rapid mapping to archive navigation. Image processing and 
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Figure 3: Unified Modeling Language-inspired EO mining context diagram depicting the basic operation of an idealized 
system. Archived images and support data are semi-automatically annotated with the help of an expert user. The final 
user queries the annotated version of the archive via a semantic interface. 



analysis, data mining, geographic data management, object detection and recognition methodologies 
are relevant, yet need to be adapted and combined in order to allow characterizing vast volumes 
of unknown — and at metric scales at least essentially uncategorizable — content. System 
structures and algorithmic choices vary significantly as well, from the considered signal characteristic 
primitive feature descriptors to data indexing approaches, from strategies for the optimization of 
specific needs for efficient storage access to supervised semantic label learning and ontology 
management. Yet in general we can assume that, in terms of functionality, an EO mining system 

• at ingestion time, automatically characterizes the input items based on a set of characteristics 
of their data and metadata contents that provide a basis — the primitive feature descriptors 
— for all further analysis; 

• semi-automatically annotates the files in an EO archive based on their content and metadata. 
Annotation happens either in terms of semantic classes (e.g. 'bridge', 'house') or in 
terms of application-independent content descriptors ('spectral', 'shape') allowing feature 
synthesis (e.g. mixing spatial/geographic and content descriptors) and data fusion (e.g. 
existing geographical maps with EO products); 

• indexes input item sets, building data structures that allow an efficient representation of 
their content for the content discovery, annotation and retrieval aims of point 3 above. 

The above points have consequences that are ubiquitous on the design and evaluation of EO 
mining systems. For instance, it has to be noted that all considerations related to efficiency in the 
remainder of this paper refer to query performance rather than optimized storage, as is typical in 
OLAP schemes (Chaudhuri and Dayal, 1997). 

3.1. Abstract system model 

According to Manning et al. (2009), the Information Retrieval (IR) problem can be stated as 

the maximization of a certain utility that ranks visual documents for a user in a specific context. 

Describing (long term) user interests about visual information is crucial to the performance of 

IR systems. To that end, relevance can be expressed explicitly via a rating scale. For example, 
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the binary scale {-1,+1) can be used to express "dislike"/"like" or "rejection"/"acceptance" 
preferences. Alternatively, a "five stars" scale such as the one used by Amazon (http : //www . 
cimazon.com/) allows the users to give more detailed degrees of appreciation. A Bayesian 
probability scale [0, 1] can be used to express degrees of belief in the statement "data archive 
item X is interesting to user Y in the operative and spatio-temporal context Z" (Jaynes, 2003). 
Again in Bayesian terms, semantic analysis and annotation in the context of EO data mining can 
therefore be understood as the procedure of finding the semantic scene elements / the labels that 
best describe a given dataset in a given operative scenario. 

We note that this way of expressing the characteristic functionality of an EO mining tool 
relates its operation to that of a scene understanding tool performing inference and estimation on 
the inversion of a direct data acquisition model. 

When models for the scene and the data become so complicated that the inversion becomes 
intractable, a common solution is a divide-and-conquer approach in which simpler intermediate 
description levels are introduced in the modeling. The scene S is then assumed to be related to 
the data D through more levels of intermediate simpler models F,- that are logically linked 

5 —> Fi —>■■■—> F„ —> D direct modelling 

D ^ Fn -* ■ ■ ■ ^ Fi — » 5 undestanding and retrieval 

The learning is in this case performed across levels to derive a conclusion about the underlying 
scene. 

The generic model corresponding to the treatment given for instance in the content-based 
image retrieval review paper by Smeulders et al. (2000) can be adapted to the context of EO and 
represented as 



D^N 



O^F 
G 



I^C^S (1) 



where 
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• D represents the available data 

• A^ represents a normalized version of D, ready for ingestion 

• O represents some form of partitioning of A' 

• F describes a set of primitive features for the objects in O 

• G is a set of signs directly extracted from the normalized data A^ 

• / is a joint indexing of the features F and the signs G 

• C represents a generic abstract classification of the data represented by the index / 

• 5 is a description of the original scene content in terms of semantic classes. 

The way this model architecture translates into algorithmic components is determined by the 
system architecture. 

3.2. Architectural options 

A general decomposition of a theoretical query process is depicted in figure 4. Archived 
products can be subject to weak and strong segmentation processes as well as to a simple 
partitioning from which regions are extracted. These regions are then subject to primitive feature 
extraction, an unsupervised data analysis step that generates signatures in metric spaces that 
express signal characteristics. Well-known scene elements ("signs") can be handled by direct 
detection and characterization. In the case of EO, these signs correspond to characteristic elements 
of specific interest such as for example road networks and simple classes of buildings such 
as silos. Concurrently, a preliminary selection of an archive collection of interest and of an 
appropriate system configuration can lead to the specification of a query. This query is formulated 
in such a way as to be usable for computing similarities in terms of the primitive descriptors to the 
analyzed archive datasets. The ranked results can then be manipulated by supervised techniques 
(e.g. including relevance feedback supervision loops) in order to be used to synthesize labels that 
can later be used to enrich the archive contents with semantic descriptions. 
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Figure 4: Idealized query process decomposition into processing modules and basic operations based on an adaptation 
of Smeulders et al. (2000). 
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While the above represents a general view of the operation of an EO mining tool, most of 
the systems are based on a broad subdivision between an ingestion component that analyzes the 
data in an autonomous manner (data discovery and normalization, primitive feature extraction, 
indexing), a learning component that is able to link the primitive feature information with semantic 
classes (supervised labeling) and a query processing system that computes image-to-label and 
pixel-to-label distances. 

The way these main stages are implemented and connected with each other in actual systems 
defines their high level architecture. 

KEO (Datcu et al., 2003) is composed of a number of separate servers with SOAP interfaces 
for much of the communication both among them and with the user interface. System web 
services and interfaces are orchestrated by the Oracle BPEL Process Manager, to ensure the 
correct data flow between modules (Munoz and Datcu, 2010). 

GeoBrowse (Marchisio et al., 1998) is based on abstract services and distributed objects. 
Its operation is based on the functionality of an object-relational database management system 
and of a scientific problem solving environment, S-PLUS. Communication between its various 
components can be established across platforms and the Internet. 

Alternative approaches are also represented. The RBIR system in Tobin et al. (2006), for 
instance, breaks down into three components: (a) a software agent-driven process that can 
autonomously search through distributed image data sources to retrieve new and updated information, 
(b) a geo-conformance process to model the data for temporal currency and structural consistency 
to maintain a dynamic data archive, and (c) an image analysis process to describe and index 
spatial regions representing various natural and man-made cover types. Again, the different 
components are interconnected by web services with well specified interfaces. 

Moving from architectural descriptions to algorithmic choices, we now start analyzing in 
detail the dififerent choices taken with regards to specific elements of an EO mining system. With 
respect to Smeulders et al. (2000), we prefer conducting the review in reverse order, starting 
with the intended final user operations, since we hope this to better clarify the different possible 
tradeoffs. 
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4. Query specification and processing 

Different approaches exist regarding how a data mining tool for EO should operate from 
the point of view of the final user with the aim of fulfilling the objectives of retrieving datasets 
of interest and characterizing them introduced in section 1. In Peijun et al. (2005), five retrieval 
patterns are proposed for EO including template-based, attribute-based, metadata-based, semanteme-based 
and integrated retrieval. 

1. Retrieval based on metadata (e.g. aquisition time, swath localization, sensor type). Although 
this paradigm is not the central subject of this review, it needs to be understood as a 
powerful basis for effective browsing and retrieval, e.g. contributing to methodologies 
such as those related to faceted search (see below). 

2. Retrieval based on the explicit specification of query attributes. In this approach, the query 
details a series of relevant attributes that can be extracted from the content of the data items 
in the searchable base. Examples might include both application-independent (e.g. color 
intervals) and specific descriptors (e.g. Normalized Difference Vegetation Index - NDVI 
values). 

3. Retrieval based on a template. A user-provided sketch of the geometry of the elements of 
interest or a (possibly multi-polygon) relevance mask on a set of existing image items or 
even full image examples are provided as a query description. A query analysis subsystem 
analyzes the templates generating a description that can be generalized and compared with 
corresponding descriptions extracted at ingestion from the items in the searchable data 
base. 

4. Retrieval based on semantemes, or minimal distinctive units of meaning, as opposed to 

sememes, the lower-level units of meaning carried by a morpheme that can be considered 

at the basis of the former query specification approaches. According e.g. to Smeulders 

et al. (2000), a semantic gap needs to be taken into account representing the lack of 

coincidence between the information that one can extract from the visual data and the 

interpretation that the same data have for a user in a given situation in linguistic, keyword-based, 

contextual terms. The semantic gap has its source in the fact that in content-based retrieval 
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is that the user seeks semantic similarity, but the database can only provide similarity by 
data processing. The challenge for image search engines — especially for those designed 
to operate on a broad domain — is to tailor the engine to the narrow domain the user has 
in mind via specification, examples, and interaction, effectively creating the relationship 
between semantic information and quantitative descriptions of EO signals. 
5. Integrated retrieval approaches, in which the complementary strenghts of the above methodologies 
can be combined in order to express complex information needs. 

4.1. Interaction patterns 

In practice, these query specification methodologies map to elements of the user interface 
and interaction patterns. 

• Query by textual semantic label: the user inputs a textual description of the scene contents 
that are being sought. 

• Query by similarity: the user provides visual examples of the elements being queried. 
The examples can be pixel-based (the user interface allows single clicks to be provided), 
region-based (the user can draw polygons around areas of interest) or tile-based (the input 
products are cut into image tiles with limited size, each of which can be used as an 
example). A further possibility is query-by-sketching, in which the user draws a simple 
outline of the elements of interest. The examples can belong to a single class or to multiple 
different classes that are considered contextually. Furthermore, some systems allow the 
user to provide both positive and negative examples for each class. 

• Metadata-based faceted search: A faceted classification system combines data search and 
browsing by classifying each information element along multiple explicit dimensions: the 
user can specify complex queries on the parameter space by applying multiple filters to 
explore the query result collection. For instance, a dataset might be reduced to a small 
set of interesting items by interactively exploring the results of filtering it with multiple 
metadata- (e.g. GeoTIFF files only) and content-based (e.g. only products containing 
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urban areas) criteria. This search paradigm might be implemented in a tree-structured 
browsing interface showing a synthetic statistical description (e.g. the current cardinality) 
of the currently selected product set (Ben- Yitzhak et al., 2008), while allowing the user 
to further filter it, as in the Amazon "Browse Books" catalogue interface (http://www. 
amazon. com/). 

Actual systems mix the different approaches in order to allow users to express the semantics of 
the applicative domains they focus on. 

KIM/KEO (Schroder et al., 2000; Datcu et al., 2003) pre-analyzes all input products, yet the 
user can in a later phase interactively define new cover classes of interest based on positive and 
negative example product regions. The classification is only used to provide visual feedback in 
the training phase. The obtained semantic labels are naturally extended to the whole archive, used 
for ranking search results and to generate classification maps on selected products. Systematic 
processing for classification map generation on large dataset is also supported. 

PicSOM (Molinier et al., 2007a) implements query by visual examples, with a relevance 
feedback loop for the interactive learning of user-level semantics. 

GeoIRIS (Shyu et al., 2007) supports query by example (possibly limited by region surrounding 
geographic and anthropogenic features), object and multiobject spatial relationship queries as 
well as semantic queries to answer information needs such as "given a query image, show me 
a set of satellite images that depict objects and spatial relationship that are similar to those in 
the query and that are located within a certain radius of a landmark." Klaric et al. (2006) and 
Shyu et al. (2006, 2007) propose bridging the semantic gap by a three-step process composed of 
data transformation (each continuous feature is partitioned into multiple discrete ranges that are 
meaningful for a specific semantic), mining associations (association rules map feature intervals 
into semantic categories) and semantic modeling in which crisp intervals from association rules 
are replaced by sigmoid functions (Barb and Shyu, 2007). 

The CNM-DM component (Garramone, 2009) allows users to query images based on contents 

via coverage percentages for a pre-defined set of thematic maps for every ingested product, 

in an annotation Data-Base (DB) that is made easily accessible by faceted search interfaces 
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(Ben- Yitzhak et al., 2008). 

Presented in Marchisio et al. (1998); Marchisio and Cornelison (1999); Marchisio et al. 
(2000), GeoBrowse is based on the abstract services and distributed objects paradigm. GeoBrowse 
consists of a Graphical User Interface, an object-relational database management system, and a 
scientific problem solving environment. Each of these can reside on a separate platform. The 
system provides support for intelligent or "content based" queries on large databases of remotely 
sensed images and incremental and random access to 3-D volumes of multispectral data from 
different sensors without the added overhead of multiple storage. GeoBrowse provides the user 
with the ability to determine and test the limitations of remote sensing parameters and models 
by providing alternative views of uncertainties arising from extrinsic factors. The scientific data 
mining environment is provided by S-PLUS, a commercial implementation of the S statistical 
programming language. A Graphical User Interface (GUI) offers database browsing capabilities 
which complement the functionality of the information retrieval engine through visual indices 
and a movie player unit. 

Connecting an EO mining system directly to applicative tools allows users to deal with 
archive- wide data analysis for objectives such as large dynamic rapid mapping and multi- temporal 
scene and sensor characterization while at the same time leveraging the pattern discovery abilities 
and the large data access capacity of the system. Durbha et al. (2008) propose that image 
information mining systems are standardized in terms of OGC specifications and in describing 
the IIM framework in an OGC perspective. This would facilitate interoperability with several 
existing OGC web services and foster the clear separation of the business logic layer and presentation 
layer. OGC standardization efforts related to geospatial data servers and processing components 
in particular promise making interoperability among system components a reality (Li et al., 
2007). KEO implements interfaces to and functionalities providing OGC compliant web services 
as both an information source and as a way to distribute the data after semantic processing. 

4.2. Supervised learning &- semantic modeling 

In order to face the potentially ambiguous meaning of image structures depending on their 

contextual understanding, especially in the case of high resolution remote sensing, in semantic 
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search engines based on a hierarchical information model of satellite image contents a supervised 
learning step is used for semantic modeling, to translate the provided examples into generalized 
rules for retrieval. 

The learned categories group and memorize the semantics of image structures, facilitating 
their recognition in various contexts. Furthermore, the generation of categories helps learning 
from a small training data set (i.e. image examples); thus, the method is useful for the exploitation 
of very large data volumes, optimizing the human machine communication. 

In the naive Bayesian approach, the assumption is made that the input features — variables 
belonging to the same level — are mutually independent and identically distributed (i.i.d.). 
This assumption is not necessarily justified, yet, naive Bayes results in a simple approach, with 
clear semantics, to representing, using, and learning probabilistic knowledge. It has often been 
shown that naive Bayes rivals, and indeed outperforms, more sophisticated classifiers on many 
datasets (??), especially when feature selection / synthesis strategies are put into place in order to 
better adapt to the i.i.d. assumption. KIM (Datcu et al., 2003) exploits Bayesian classifiers and 
Dirichlet models for defining classes of interest for users. It manages a taxonomy of semantic 
labels defined on the hierarchy of signal-based labels. KIM uses a Bayesian meta-probabilistic 
measure to link images to defined labels, whereas GeoIRIS (Shyu et al., 2007) employs sigmoid-shaped 
functions to the same end. Aksoy (2007) uses Bayesian classifiers to compute the final classification 
maps using region level information. To be able to use Bayesian classifiers, different region-based 
features such as statistics and shape features are independently converted to discrete random 
variables using the K-Means algorithm for vector quantization. 

In Lienou and Campedel (2009); Lienou (2009), basic features like means and variances are 
computed on a window around each pixel, in each spectral band. These descriptors are then 
vector-quantized using a K-Means algorithm. Semantic annotation of satellite images using the 
Latent Dirichlet Allocation (LDA) model. The model combines a step of supervised Maximum 
Likelihood classification of patches of the large image to be annotated, and the integration of the 
spatial information between these patches by considering a partial overlap among them. 

In Lienou et al. (2010), given a training set of images for each concept, learning is based on 
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the LDA model. This hierarchical model represents each item of a collection as a random mixture 
of latent topics, where each topic is characterized by a distribution over words. The LDA-based 
image representation is obtained using simple features extracted from image "words". The 
capability of the LDA model to assign probabilities to unseen images is then used to classify the 
patches of the large image into the semantic concepts, using the maximum-likelihood method. 

Support Vector Machines (SVMs) provide the basis for a number of systems. Mountrakis 
et al. (2011) review remote sensing implementations of SVMs and compare them with methods 
from maximum likelihood classifiers to neural networks in a wide range of applications from 
coal reserve detection to urban growth monitoring. 

In Shah et al. (2004), a learning phase is applied at this stage that associates the middle level 
descriptors to the concepts in the higher-level ontology by means of an SVM (Vapnik, 1999) 
method. These associations are grouped into models specific to a semantic class and used for 
querying. 

In Li and Narayanan (2004) land cover information corresponding to spectral characteristics 
is identified by supervised classification based on support vector machines with automatic model 
selection, while textural features characterizing spatial information are extracted using Gabor 
wavelet coefficients. Within identified land cover categories, textural features are clustered to 
acquire search efficient space in an object-oriented database with associated images stored in an 
image database. 

Costache and Datcu (2007); Costache (2008) propose Bayesian inference to learn categories 
and a Support Vector Machine (SVM) classifier to assign semantics. The approach is enhanced 
with learning / unlearning functions. Costache et al. (2006a) present a probablistic extension 
of SVMs in a Bayesian framework that allows complementing them with memory simulation 
mechanisms for machine learning purposes. 

Support Vector Machines are combined with active relevance feedback in Ferecatu et al. 
(2005); Ferecatu (2005); Ferecatu and Boujemaa (2007). 

Decision trees are also exploited for structur EO data mining. Aksoy et al. (2009) show how 
decision tree classifiers can be learned with alternative (surrogate) decision nodes and result in 
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models that are capable of dealing with missing data during both training and classification to 
handle cases where one or more measurements do not exist for some locations. Aksoy and Cinbis 
(2010) compute the degrees of satisfaction of the extracted spatial relationships between objects 
that satisfy user-specified attribute criteria. The objects are ranked according to a combined 
measure (e.g., product, sum, and weighted sum) that involves the confidence of detection, the 
attribute values, and the spatial constraints. 

Bordes and Prinet (2008) use a bag-of-words representation of textons while taking into 
account spatial information. A generative probabilistic modeling of the distribution of textons is 
proposed. The parameters of the mixtures components are estimated using a Expectation-Maximization 
algorithm (Moon, 1996), and the number of classes in a database estimated by Minimum Description 
Length (Rissanen, 1985). 

Koperski and Marchisio (2000); Koperski et al. (2002b) describe a system for interactive 
training of models for semantic labeling of land cover. The models are build based on three levels 
of features: 1) pixel level, 2) region level, and 3) scene level features. A Bayesian algorithm 
and a decision tree algorithm are developed for interactive training. The Bayesian algorithm 
enables training based on pixel features. The scene level summaries of pixel features are used 
for fast retrieval of scenes with high/low content of features and scenes with low confidence 
of classification. Graphical tools for the exploration of decision trees allow insight into the 
interaction of features used in the construction of models. 

4.3. Iterative query specification 

A common approach allows users to specify examples for the query and to evaluate the 
effectiveness of the currently defined one in successive iterations. 

Relevance feedback techniques are used in tools like KIM (Datcu et al., 2003) and PicSOM Molinier 

et al. (2007a) to improve retrieval efficiency. Blanchart et al. (201 1) take into account semi-supervised 

methods by mixing an auto-annotation component with a category search engine which combines 

generic image class search and object detection. The proposed concept relies thus on complementary 

elements: an auto-annotation component, a generic category search engine and an object detection 

tool. 
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Alber et al. (2001) shows that a triangle inequality search technique applied to a relevance 
feedback retrieval algorithm can significantly speed up the search for and retrieval of physical 
events of interest in large remote-sensing databases. An improvement in retrieval speed is 
illustrated using hurricane queries applied to the multispectral GOES weather satellite database. 

Costache et al. (2006b) present a categorisation-based relevance feedback search engine for 
EO images repositories. The developed method is based on SVMs: the process starts with a query 
phase in which the user is selecting among a randomly machine-generated image sequence one 
which best describes his interest. The selected image is then used by the system for retrieval 
purposes in the following way: based on a measure of similarity (e.g. Euclidean distance) the 
system performs a ranking of the images and returns the top and bottom images. The returned 
images are then labelled by the user as relevant/irrelevant and are used to train an SVM classifier. 
The systems performs again a ranking but this time, based on the absolute value of the distance 
function, in ascending order. The top images with the smallest absolute value of distance function 
are selected to be shown to the user at the next step. They are the most ambiguous images as 
they have a very small decision function value. In this way one step after another the separation 
surface between the relevant and irrelevant images is better traced. 

Non-binary graded relevance feedback is considered by a few authors. Most of existing 
methods for annotating semantic meaning to geospatial images are trained using binary feedback 
from users. Such approaches may lead to suboptimal models especially due to the fact that 
semantic relevance of images is rarely a binary problem. Barb and Shyu (2010a), Barb and Shyu 
(2010b) present an algorithm to link low-level image features with high-level visual semantics 
using graded relevance feedback from image analysts. This linkage is done using flexible fuzzy 
possibility functions that mathematically model the existence of visual semantics in new images 
added to the database. 

A number of papers (Ferecatu and Boujemaa, 2007; Tuia et al., 2009) focus instead on active 
learning for improving remote sensing content based retrieval: the learning algorithm is able to 
interactively query the user to obtain the desired outputs at new data points. 

The same approach is used by Rital et al. (2008). They propose to adapt a classical multimedia 
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CBIR approach to EO images: they are cut into small images from which a vectorial signature 
is extracted. Then an active learning-based retrieval algorithm is applied in order to profit by 
the human expertise. The result of each query can be stored in a memory using both keywords 
and classifier model. The system is evaluated using a small labeled database corresponding to a 
typical land cover classification task. 

In Li and Bretschneider (2007) a stepwise retrieval scheme is adopted to balance effectiveness 
and efficiency. The semantics are used to retrieve a set of candidate images that are related to the 
estimated concepts of the user Then, the similarity between the query and the candidate images 
found in the first stage is measured by integrated region matching. 

4.4. Visualization and visual analytics 

The topic of visualization represents an issue with respect to the main data processing in EO 
mining whose importance cannot be overstated: properly addressing the human visual system of 
both expert and basic users in the diagram in figure 3 requires detailed consideration. 

The goal of visual enhancement methods — either by image processing techniques or by 
selecting specific spectral bands — is to maximize the response in the human visual system and 
increase the saliency of the object / area of interest. 

Bratasanu et al. (201 1) propose a spectral band discovery methodology for advancing multispectral 
satellite image visual analysis. The paper describes an interactive technique to discover the 
optimum combination of three spectral features of a multispectral satellite image that enhances 
visualization of target classes / objects. 

The inclusion of a Self-Organizing Map (SOM) is one of the central elements of PicSOM Molinier 
et al. (2007a,b) and of its user interface and key algorithmics. In it, after a SOM depicting 
and structuring the data is trained, the user can visually query the database and the system 
automatically finds images similar to those selected. 

Chen et al. (2008) develop a visual analytics approach that leverages human expertise with 
visual, computational, and cartographic methods to support the application of visual analytics 
to relatively large spatio-temporal, multivariate data sets. It develops and apphes a variety of 
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methods for data clustering, pattern searching, information visualization, and synthesis, focusing 
on combining both human and machine-based approaches to discover hidden information. 

Although not directly interested with EO data but rather with geographic data, Guo (2003) 
describes a human-centered exploration environment, which incorporates a coordinated suite 
of computational and visualization methods to explore high-dimensional data for uncovering 
patterns in multivariate spaces. It includes interactive feature selection and hierarchical clustering 
methods as well as a suite of coordinated visualization and computational components centered 
around the above two methods to facilitate a human-led statistical data exploration. Guo et al. 
(2006), Guo and Gahegan (2006) integrate computational, visual, and cartographic methods 
to develop a geovisual analytic approach for exploring and understanding spatio-temporal and 
multivariate patterns. The developed methodology and tools can help analysts investigate complex 
patterns across multivariate, spatial, and temporal dimensions via clustering, sorting, and visualization. 
Specifically, the approach involves a SOM, coordinate plots, several forms of reorderable matrices, 
a geographic multiple display, and a 2-dimensional cartographic color design method. The 
coupling among these methods leverages their independent strengths and facilitates a visual 
exploration of patterns that are difficult to discover otherwise. 

4.5. Query processing 

The query execution process takes in input a set of query items and returns a set of ranked 
results. While execution performance is the primary concern, important dififerences arise with 
respect to the query specification process that subsequently determine the characteristics of all 
system components that prepare the data for this step. Query execution of course drastically 
depends on the query formulation strategy. 

Tools such as the CNM-DM component (Garramone, 2009) only output thematic maps based 
on pre-defined classes and coverage percentages that are fed into an attribute DB. 

Most other systems allow users to interactively define new elements of interest instead. A 

first approach is to allow query by class / query by value based on vector quantization codewords. 

Vellaikal et al. (1995) use vector quantization codewords as the remote sensed image features for 

content based retrieval. Different distortions measures are considered to enhance the performance 
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of the codewords as "content descriptors" including classification accuracy. Both query by class 
and query by value are implemented. 

KIM (Datcu et al., 2003) uses a relevance feedback mechanism for updating a Bayesian 
network that is used to perform data classification and retrieval tasks. 

In the GeoBrowse system, decision tree classifiers are used (Aksoy et al., 2004) for interactive 
learning of land cover models and mining of image archives: they can operate on both numerical 
(continuous) and categorical (discrete) data sources, and do not require any assumptions about 
neither the distributions nor the independence of attribute values. 

A methodology to execute complex queries by the integration of an inference engine is 
considered by Durbha et al. (2005a) for I^KR. The paper maintains that pursuing the semantic 
web model for semantic annotation of remote sensing data archives provides attractive alternatives 
to the traditional methods of information integration and retrieval. It builds upon semantic web 
technologies and combines them with pattern recognition and machine learning techniques to 
develop a framework for semantics driven retrieval of knowledge from EO data archives. At 
the heart of the framework is a middleware for ontology brokering that provides support for 
the development of application level ontologies that represent the concepts of the information 
sources. The Ontology Web Language is used to build them. Domain-specific ontologies help 
to define concepts in a finer granularity. These fine-grained concepts then allows to determine 
specific relationships among features (e.g., shape, texture, color) in images that may be used to 
classify those images. Durbha and King (2005) employs an unsupervised segmentation algorithm 
to extract homogeneous regions and calculate primitive descriptors for each region based on 
color, texture, and shape. They perform an unsupervised classification by means of a kernel 
principal components analysis method, which extracts components of features that are nonlinearly 
related to the input variables, followed by a support vector machine classification to generate 
models for the object classes. The assignment of concepts in the ontology to the objects is 
achieved automatically by the integration of a description logics-based inference mechanism, 
which processes the interrelationships between the properties held in the specific concepts of 
the domain ontology. In this line, Durbha et al. (2005b, 2007) add a methodology for domain 
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specific qualitative spatial reasoning in coastal wetlands. In Durbha et al. (2007), a support vector 
machines-based classification is applied for generating predictive models. Durbha et al. (2010) 
propose a set of feature selection and feature transformations based on a wrapper-based genetic 
algorithm approach. A support vector machine classification is applied for generating predictive 
models for those land-cover classes that are important in a coastal disaster event. 

Ruan et al. (2006) propose a framework based on domain-dependent ontology to perform 
semantic retrieval in image archives. Homogeneous regions in the data products are described by 
high-level concepts depicted and organized by a domain specific ontology. Interactive learning 
techniques are employed to associate regions and high-level concepts. These associations are 
used to execute queries. Additionally, a reasoning mechanism integrating an inference engine 
enables mining the inter-relationships among domain concepts and their properties to satisfy user 
requirements. An ontology is used to provide a sharable and reusable concept set as infrastructure 
for high level extension such as reasoning. 

Datta et al. (2006) proposes a two-stage architecture for automatic retrieval of satellite image 
patches. Semantic categorization is done by a learning approach involving the two-dimensional 
multi-resolution hidden Markov model. Items that do not belong to any trained category are 
handled using an SVM-based classifier. 

4.5.7. Data management 

Effectively exploiting data storage in order to maximize the potential of EO mining systems is 
a central issue for pre-operational and operational systems. Different approaches are considered 
in the literature with respect to this point. 

While a number of systems such as KIM (Datcu et al., 2003) exploit relational DBs essentially 
as data storage facilities, Shyu et al. (2006) exploit relational queries directly by representing in 
them mining association rule intervals. 

NoSQL DBs (Xiao and Liu, 201 1) promise scalability, network partitioning, replication and 
higher performance than traditional relational systems for remote sensing image databases. 

An approach to spatial On-Line Analytical Processing (OLAP) based on data cubes is presented 

in Rivest et al. (2005). The paper specifically focuses on interactive spatio-temporal exploration 
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of data via geovisualization/animation. 

ASI's CNM-DM (Garramone, 2009) and prototype systems at ESA such as the User Services 
Next Generation tool already provide faceted content-based search based on object-oriented 
databases. They exploit a local relational Data-Base Management System (DBMS) for temporary 
object serialization, feeding data into an object oriented DB for attributes that are then made 
available for faceted search. 

The RBIR system described by Tobin et al. (2006) exploits XML-based DBs instead for 
metadata and content-based searches. It describes the implementation of a binary decision tree 
of the image features based on k-dimensional (kd-) tree methods (Arya et al., 1998). The 
accuracy of the system is selectable as a trade-oflF between nearest neighbor performance and 
computational efficiency. Through this approach, retrieval efficiencies on the order of five seconds 
for 100,000 indexed images can be demonstrated (Tobin et al., 2002). 

4.5.2. Index generation and maintenance 

Indexing is meant to improve the speed of data retrieval operation by building summary 
representations of the content of the database in terms that exploit the inherent organization of 
the data to allow very fast query execution procedures: the goal of indexing is to organize the 
image data (e.g., filenames, features, indexing codes, etc.) in the database such that a ranked list 
of nearest neighbors can be efficiently retrieved in response to a query without performing an 
exhaustive comparison to all the records in the database. 

In systems such as KIM (Datcu et al., 2003) no indices are stored in the DB. The ranking 
phase operates sequentially on all records by dynamically computing Bayesian estimates of 
query-to-tile match based on the selected primitive features. 

Whereas an exhaustive nearest-neighbor search of the n vectors (i.e., images) in the database 
would imply of 0{n) computations, a kd-tree approach involves a complexity 0(log2(n)). 

In GeoIRIS (Shyu et al., 2007) two indexing mechanisms are used for tile- based and object-based 

query methods. Indexing of continuous valued features is done using the entropy balanced 

statistical (EBS) k-dimensional (k-D) trees, and indexing the binary-valued features is performed 

with the entropy balanced bitmap (EBB) tree (Scott et al., 201 1), which exploits the probabilistic 
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nature of bit values in automatically derived shape classes. Tile-based indexing provides access 
into localized areas of similar features. Object-based indexing includes both individual objects 
and spatial configurations of multiple objects. 

Column-based and raster DB systems (Ivanova et al., 2007) (Widmann and Baumann, 1998) 
attempt at automatic record optimization and autonomous index generation. Altough the results 
largely improve on sequential scan in traditional relational DBMSs, more advanced implementations 
(e.g. based on k-d trees) are needed (Kao et al., 1998) for optimal results. 

5. Ingestion 

The first necessary step before data analysis is the discovery of the data and its normalization 
so that all subsequent phases of the analysis process can operate eff'ectively. This functionality 
corresponds to crawlers in web information retrieval engines. 

KEO (Schroder et al., 2000; Datcu et al., 2003) is directly connected to specific DFD and 
ESA feeds and rolling archives and continuously updates its contents respectively with MerisRR 
and TerraSAR-X data. 

In a parallel concept, Bingham et al. (2009) takes the concept of Really Simple Syndication 
(RSS) feeds, for delivering regularly changing web content, and extend this to represent a stream 
of data granules and deliver regularly changing Earth science data content. It envisages filtering 
content based on the metadata of a feed in order to identify granules of interest based on user-defined 
criteria. 

5.7. Dataset tiling 

In terms of Smeulders et al. (2000), a few possible ways exist to decompose images in order 
to analyze their contents, each one corresponding to a different class of content descriptors. 

• No decomposition: sign extraction. 

• Resolution reduction: salient features. 

• Generic tiling: generalistic features. 
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• Segmentation: region features. 

• Object extraction: object-based descriptors. 

The effects of sensor resolution are especially important in determining which of the approaches 
provides the most interesting results. 

A tiling approach similar to the one in Rital et al. (2008) is used in the PicSOM system (Molinier 
et al., 2007a). Here, query by visual examples, relevance feedback and ranking of images rely 
on the standard multimedia CBIR approach related to cutting satellite images into small images 
(imagelets). Most systems consider instead geographically-inspired tiling grids. 

Trias-Sanz et al. (2008) review several existing colour space transformations and textural 
features for the segmentation of high-resolution multispectral aerial images with a hierarchical 
segmentation algorithm. A method to quantitatively evaluate the quality of a hierarchical image 
segmentation based on Pareto optimality is presented, and the behaviour of the segmentation 
algorithm for various parameter sets is explored. 

Guo et al. (2009) proposes a semantic-aware two-stage wavelet-based image segmentation 
approach, which preserves the semantics of real-world objects during the segmentation process. 
The system is specifically aimed at high resolution remote sensing image retrieval. 

Gueguen et al. (201 1) presents an interactive image information mining tool handling millions 
of structures in a scene. The learning process is incremental, incorporating new training samples 
at low computational cost. Input images are first segmented in multi scale blobs by segmentation 
of Differential Morphological Decomposition, and then are characterized by spectral and shape 
features. Both cluster and SVM based classifiers are compared in terms of accuracy, where 
the accuracy is the complement of the pixel based probability of errors. Classifier training is 
performed incrementally in both cases, benefiting from a global classification visualization at 
each step. 

Shah et al. (2004) employs an unsupervised segmentation algorithm to extract homogeneous 
regions and calculate primitive descriptors for each region based on color, texture and shape. The 
primitive descriptors are described quantitatively by middle level object ontology. 
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5.2. Primitive features and signs 

Ingestion operates by analyzing the different information sources trying to unsupervisedly 
characterize them, often in application-independent manners. EO mining systems operate on 
primitive signal features extracted from the original data. Their performance depends crucially 
on them and on their adaptation to specific mining objectives. Generalist primitive feature 
extractors are used to make application-independent image mining possible. These can then 
be complemented by vertical application-specific feature extractors dedicated to specific targets, 
corresponding to "sign" descriptors in Smeulders et al. (2000). 

As per Smeulders et al. (2000), all systems consider first a primitive feature extraction step 
that transposes the image data into another spatial data array. The different descriptor extraction 
methods (e.g. color, local texture, local geometry) may be characterized in general by: 

fix) ^ go i(x) 

where i(x) is the image, element of image space /, g is an operator on images, and the resulting 
image field is given by fix). Computational parameters of g may include the size of the neighborhood 
around x to compute f{x) or a homogeneity criterion when the size of the patch to compute f{x) 
depends on the actual data, as in Soccorsi et al. (2006), for example. The purpose of this signal 
processing in image retrieval must be to enhance aspects in the image data relevant to the query 
and to reduce the remaining aspects. 

The aspect of dealing with invariance as a tool to deal with accidental distortions in the 
information introduced by the sensory gap is often handled by primitive classification. Again 
according to Smeulders et al. (2000), the aim of invariant descriptions is to identify objects, no 
matter from how and where they are observed, at the loss of some of the information content. If 
two objects f, (or two appearances of the same object) are equivalent under a group of transformations 
W, they are in an equivalence class 



ti ~ t2 ^ 3w e W : tx -w o ti 
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A property / of f is invariant under W if and only if /, remains the same regardless the unwanted 

condition expressed by W, 

w 

Again as noted in the paper cited above, the degree of invariance, that is, the dimensionality of 
the group W, should be tailored to the recording circumstances. The aim is to select the tightest 
set of invariants suited for the expected set of non constant conditions. 

We classify the primitive extraction into global features, salient features, object features and 
signs according to Smeulders et al. (2000). 

5.2.7. Signs 

The ability of an IR system to effectively incorporate and exploit an external primitive feature 
extractor for search in large archives is a central factor in defining its adaptability and hence its 
real-world applicability. 

The KEO system (Datcu et al., 2003) includes tools to allow users specify primitive feature 
extractors dedicated to specific target classes. 

Environments such as ENVI/IDL (Canty, 2007; Research Systems, 2003) include large repositories 
of image and object analysis modules. In particular, the ENVI EX system provides feature 
extraction, classification, orthorectification and change detection functions. An integration with 
the ArcGIS platform allows users to integrate image analysis processes into GIS models via 
high-level programming languages such as IDL. 

The eCognition image analysis software is introduced by Benz et al. (2004) with the principal 
strategies of object-oriented analysis behind it, discussing how the combination with fuzzy methods 
allows implementing expert knowledge and describing a representative example for the proposed 
workflow from remote sensing imagery to GIS. 

Similarly, QGIS (http: //www. qgis.org/), gvSIG (http://gvsig.org/) and GRASS 
(http : //grass . osgeo . org/) offer an open-source alternative (Neteler et al., 2006) for integrated 
GIS and remote sensing image analysis that includes statistical analysis tools based on low- as 
well as high-level programming environments such as R (http : //r-pro j ect . org/). 
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Inglada and Christophe (2009) describes the Orfeo ToolBox (OTB), a remote sensing image 
processing library developed by CNES that includes tools for the operational exploitation of the 
future sub-metric optic and radar images (rapid mapping, 3D aspects, change detection, texture 
analysis, pattern matching, optical and radar complementarity). A number of the algorithms 
made available in the OTB can be used as sign extractors in EO mining engines. 

5.3. Object- and region-oriented descriptors 

Object-oriented methods (Blaschke, 2010) are making considerable progress towards a spatially 
explicit information extraction workflow, such as is required for spatial planning as well as for 
many monitoring programmes. 

The exploitation of available prior knowledge maps for the extraction of important topographic 
objects, like buildings and roads, is investigated in Baltsavias (2004). The paper focuses on 
aspects of knowledge that can be used for object extraction: types of knowledge, problems in 
using existing knowledge, knowledge representation and management, current and possible use 
of knowledge, upgrading and augmenting of knowledge. An overview on commercial systems 
regarding automated object extraction and use of a priori knowledge is given. 

The GeoIRIS system (Shyu et al., 2007) supports the retrieval of tiles according to the 
spatial configuration of the objects they contain. In particular Scott et al. (2005) includes a 
method to model spatial relationships among sets of three or more objects in satellite images 
for scene indexing and retrieval by generating discrete spatial signatures. The method is highly 
insensitive to scaling, rotation, and translation of the spatial configuration. In Aksoy and Cinbis 
(2010), directional spatial relationships among objects are considered to enable object detection 
based on the properties of individual objects as well as their directional spatial relationships to 
other objects based on morphological modeling of relative-position-based spatial relationships. 
This information is incorporated into the Bayesian decision rule as spatial priors for contextual 
classification and retrieval. The directional landscapes considered can be used for image retrieval 
for geospatial intelligence. 

CBIR using shapes and topology is considered in Agouris et al. (1999) for sketch-based 

query formulation. A spatial data management system optimized for such queries and a set of 
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features describing the geometry of the sketched query shapes as well as their spatial arragement 
are presented. 

In Guo et al. (2009), to better capture semantic features for object discovery, a hyperclique 
pattern discovery method is exploited to find in the DB co-occurrence patterns representing 
complex objects that consist of several co-existing individual objects that usually form a unique 
semantic concept. In Gautama et al. (2007), spatial relations between objects are used to find 
a reliable object-to-object mapping. Graph matching is used as a flexible query mechanism to 
answer the spatial query. 

Except for sign extractors, other kinds of data content descriptors need data partitioning 
strategies. Both strong and weak segmentation can be used to this end, resulting respectively 
in saUent features and in region based descriptors. 

Even though correct identification of pixels and regions improvse the processing time for 
content extraction, manual interpretation is often necessary for many applications because two 
scenes with similar regions can have very different interpretations if the regions have different 
spatial arrangements. Therefore, modeling spatial information to understand the context is an 
important and challenging research problem. In Tobin et al. (2006), once the features of the 
image segments have been extracted, it is possible to use this feature vector as an index for 
retrieval. A geospatial clustering procedure is performed using a region growing technique to 
connect large contiguous and homogeneous segments of similar structure and texture characteristics. 
In Li and Bretschneider (2007), every image is segmented into disjoint regions using a simple 
segmentation algorithm, and the low-level features are extracted from each region. In a second 
step, the region descriptors, i.e., multidimensional vectors, are classified into a finite number of 
frequent patterns with similar appearance by using a vector quantization algorithm which creates 
a reduced representation of possible region descriptors. The images in the database are encoded 
by so-called codes based on the result of the previous quantization. 

Shape similarity is a powerful tool for query processing. Dell'Acqua and Gamba (2001) 
present a "point diffusion technique" method for shape similarity evaluation suitable for application 
to similarity-based retrieval from remotely sensed meteorological image archives, where shapes 
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are hardly defined but are still among the major features of interest. Silva et al. (2005) combine 
a region-growing segmentation algorithm on multi-temporal change images with a structural 
classifier that operates on shape descriptors for mining evolutions typically related to deforestation 
patterns. In Aiyesha (2005), from 56 full hemisphere infrared scans of the earth taken by 
Meteosat satellites, regions are extracted using region growing. After region extraction, polygonal 
approximation is applied to the region shape, and local features of the polygons are hashed to 
provide an association space. This space becomes the indexing structure through which retrieval 
takes place. Aksoy (2007) models regions using the statistical summaries of their spectral and 
textural properties along with shape features that are computed from region polygon boundaries. 
The statistical summary for a region is computed as the means and standard deviations of features 
of the pixels in that region. Multi-dimensional histograms also provide pixel feature distributions 
within individual regions. 

Relationships among regions can be considered as well. The VisiMine system (Tusk et al., 
2003; Koperski et al., 2002a; Aksoy et al., 2002, 2005) includes automatic methods for the 
extraction of topological, distance-based, and relative-position-based relationships between region 
pairs where such relationships can be successfully used for image classification and retrieval in 
scenarios that cannot be expressed by traditional pixel- and region-based approaches. Kalaycilar 
et al. (2008) describe an image representation using spatial relationship histograms that extend 
relational graphs. These histograms are constructed by classifying the regions in an image, 
computing the topological and distance-based spatial relationships between these regions, and 
counting the number of times different groups of regions are observed in the image. A selection 
algorithm produces compact representations by identifying the distinguishing region groups that 
are frequently found in a particular class of scenes but rarely exist in others. 

5.4. Spectral and textural descriptors 

Almost all of the considered systems use spectral and textural descriptors as well as local 
edges and patterns. 

GeoIRIS (Shyu et al., 2007) exploits histograms for panchromatic, grayscale RGB, and 

near-infrared data. As textural features, KIM (Schroder et al., 2000; Datcu et al., 2003) considers 
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Gauss-Markov Random Field-based descriptors (usually from a single specific channel) for both 
optical and SAR data. Advanced Bayesian reconstruction methodologies (Walessa and Datcu, 
2000) are used to cope with the specificities of SAR sensors. 

Aksoy (2007) extracts spectral and textural features for each pixel. Gabor texture features 
are extracted by filtering the first principal component image with Gabor kernels at different 
scales and orientations and using kernels rotated by imjA, n = 0, ■ ■ ■ , 3, at 4 scales resulting in 
feature vectors of length 16. Finally, each feature component is normalized by linear scaling 
to unit variance. Tobin et al. (2006) primarily exploit edge information to describe texture and 
structure to avoid performance degradation due to variation in the spectral content. The approach 
characterizes segment texture using local binary patterns (Pietikainen et al., 2000) and local edge 
patterns (Yao and Chen, 2003). The GeoIRIS system described in (Shyu et al., 2007) exploits 
texture measures based on Haralick co-ocurrence matrices. 

Shah et al. (2005, 2007) propose to perform image segmentation using color and texture 
features from the wavelet coefficients for the region-based retrieval in remote sensing image 
archives. Based on that. Shah et al. (2006) exploit Independent Component Analysis (Comon, 
1996) for feature selection/synthesis. 

Clustering issues include the possibility to consider different distance measures among n-dimensional 
multi-spectral vectors (Cerra et al., 201 1), ranging from euclidean (Keogh et al., 2004) to spectral 
angle (Kruse et al., 1993, 1999), from spectral correlation (De Carvalho and Meneses, 2000) to 
information divergence measures (Du et al., 2004). KIM (Datcu et al., 2003) uses K-Means 
clustered multi-spectral feature histograms based on either euclidean or spectral angle distance 
measures. 

A few systems consider specialized descriptors as well. In the GeoIRIS system (Shyu 
et al., 2007), specialized features are exploited to represent characteristics of linear features such 
as roads (linearity, directionality) and the scale of objects such as buildings (pixel correlation 
run-length and scale-based descriptors of object content generated from differential morphological 
profiles (Pesaresi and Benediktsson, 2001)). Furthermore, a set of features based on normalized 
grid descriptors are extracted for each object (Zhang and Lu, 2001). The histogram of forces 
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approach (Matsakis, 1998) is used to generate a spatial signature of an object configuration 
by extending the pairwise determination of spatial relationships. GeolRlS also includes an 
"anthropogenic" primitive descriptors to try and separate populated from natural areas. The 
PicSOM system (Molinier et al., 2005, 2007a) exploits as features color moments and textures 
together with NDVI and edge histograms. 

5.5. Data fusion 

Data fusion is a significant topic in its own right. Molch (2010) for instance examines the 
representation of semantic categories integrating Ikonos and Quickbird imagery in the knowledge-based 
information mining system KIM. A processing sequence is presented, which accounts for sensor-related 
differences along with an evaluation of the application of IIM technologies in operational rapid-mapping 
scenarios. 

The PicSOM system (Molinier et al., 2007 a,b), includes features adapted to man-made structures 
detection in high-resolution optical satellite images. Fusion of panchromatic and multispectral 
information can be obtained by training several SOMs in parallel (one per feature). Molinier 
et al. (2008) investigates the potential of PicSOM for quad-polarised ALOS PALSAR images. 
The system was originally developed to operate on QuickBird MultiSpectral and Panchromatic 
data. 

5.6. Multi-temporal mining 

Satellite Image Temporal Series (SITS) represent a very active objective of EO mining research: 
the extension of the change detection problem — including high geometric resolution data 
(Padwick et al., 201 1) — to long series of semi-periodic data. 

In Walter (2004), a change detection approach based on an object-based maximum likelihood 
classification of groups of pixels corresponding to objects in an existing GIS database used to 
provide training areas. 

The PicSOM system (Molinier et al., 2007a) implements original methods for both supervised 

and unsupervised change detection based on the ranking of discrimination values for imagelet 

pairs, and distance on the SOM lattice. The system is specifically evaluated for the analyzing 
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variations in the presence of buildings, differentiating true changes with respect to unsubstantial 
effects (e.g. changes in illumination). 

A seminal work is represented by Heas and Datcu (2005): a hierarchical Bayesian modeling 
of SITS information content enables uerss to link semantic interests to specific spatio-temporal 
structures. The hierarchy is composed of two inference steps: an unsupervised modeling of 
dynamic clusters resulting in a graph of trajectories, and an interactive learning procedure based 
on graphs which leads to the semantic labeling of spatio-temporal structures 

The IIM-TS system (Bovolo and Bruzzone, 2007), instead, integrates probabilistic multi-temporal 
descriptors into the KEO system. 

On a methodological level, information theoretically motivated approaches are well represented. 
Cerra et al. (2010) and Gueguen and Datcu (2007) present an algorithm based on the Information 
Bottleneck principle composed of a parameter estimation and a model selection. Two approaches 
are presented. In the first approach, each image of the SITS is segmented and the obtained 
regions are described by textural models. The Information Bottleneck method is further used to 
characterize the image segments of the SITS a spatio-temporal way. In the second method, the 
geometrical information is extracted from a temporal adjacency graph of the spatial regions, and 
the radiometric and textural information is then extracted through the Information Bottleneck 
method. This approach leads to a temporal characterization of the spatial regions of the SITS. In 
the context of compression-based analysis, Gueguen and Datcu (2008) present a Satellite Image 
Time Series coding system that produces a code in two parts: one contains the predictor and the 
statistical context tree, and the other one contains the residual error coded in an optimal way. 
This enables the creation of an index of spatio-temporal structures. 

Julea et al. (2006) proposes an approach similar to that of Mantovani et al. (2009); Natali 
et al. (201 1) and based on the use of sequential patterns for the analysis of multi-temporal remote 
sensing data. Indeed, as sequential patterns include the temporal dimension, they can be used 
for extracting frequent evolutions at the pixel level, i.e. frequent evolutions that are observed 
for geographical zones that are represented by pixels. While Julea et al. (2006) deals with 
METEOSAT and ERS satelHte images, Mantovani et al. (2009); Natali et al. (201 1) Multi-sensor 
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Evolution Analysis (MEA) is able to deal with 15 years of global (A)ATSR data (1 km resolution), 
together with 5 years of regional AVNIR-2 data (10 m resolution) to permit on-the-fly Land Use 
/Land Cover Change analysis. 

5.7. Discretization 

As mentioned above, clustering issues include the possibility to consider different distance 
measures among n-dimensional multi-spectral vectors (Cerra et al., 201 1), ranging from euclidean 
(Keogh et al., 2004) to spectral angle (Kruse et al., 1993, 1999), from spectral correlation (De Carvalho 
and Meneses, 2000) to information divergence measures (Du et al., 2004). 

KIM (Datcu et al., 2003), for instance, uses K-Means clustered multi-spectral feature histograms 
based on either euclidean or spectral angle distance measures. GeoIRIS (Shyu et al., 2007) 
exploits histograms for panchromatic, grayscale RGB, and near-infrared data. Maheshwary and 
Srivastava (2009) apply K-Means on color moments and co-occurrence matrices, then computes 
Euclidean distances and validates the approach on three LISS III + multi-spectral satellite images 
with 23.5 m resolution. 

Hybrid hierarchical approaches are also represented. Aksoy (2006) describes a hybrid hierarchical 
approach for image content modeling and retrieval. First, scenes are decomposed into regions 
using pixel-based classifiers and an iterative split- and-merge algorithm. Next, spatial relationships 
of regions are computed using boundary, distance and orientation information based on different 
region representations. Finally, scenes are modeled using attributed relational graphs that combine 
region class information and spatial arrangements. 

Still working on limited amounts of data, a novel approach is represented in Dong and 
Xiang-bin (2008) and Bedawi and Kamel (201 1), Particle Swarm Optimization is used to classify 
remote sensing data: classification rules are generated through simulating the behaviors of bird 
flocking. Optimized intervals of each band are found by particles in multi-dimension space, 
linked with land use types for forming classification rules. Its performance with respect to 
generalization issues still needs to be assessed in the case of large archives. 

Hierarchical partitioning is considered by a number of authors. Bahroun et al. (2010) propose 

a hierarchical visual thesaurus of the regions provided by a region labeling criterion based on 
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point pattern analysis into homogeneous and textured regions for boosting the object recognition. 
The labeling criterion is based on the spatial dispersion of interest points in the region validated 
on a satellite image database. 

5.8. Feature selection and ordering 

Techniques such as ICA (Comon, 1996) and PC A (Abdi and Williams, 2010) are routinely 
used to generate more efficient feature sets and to try and overcome the so called "curse of 
dimensionality". 

In Durbha et al. (2007), the relevance of a set of primitive features for a particular land cover 
class or a combination of classes is then assessed based on a wrapper-based genetic algorithm 
approach. Furthermore, an induction algorithm is used along to arrive at an optimal set of 
features. 

In Aksoy (2007), spectral and textural features are quantized and are used to train Bayesian 
classifiers with discrete non-parametric density models. An iterative split-and-merge algorithm 
is used to convert the pixel level classification maps into contiguous regions. Then, the resulting 
regions are modeled using the statistical summaries of their spectral, textural and shape properties. 
To simplify computations and to avoid the curse of dimensionality during the analysis of hyper-spectral 
data, Aksoy (2007) applies Fishers linear discriminant analysis that finds a projection to a new 
set of bases that best separate the data in a least-squares sense. It applies principal components 
analysis that finds a projection to a new set of bases that best represent the data in a least-squares 
sense. The top 10 principal components are kept instead of the large number of hyper-spectral 
bands. Finally, discrete variables and a non-parametric model in the Bayesian framework are 
considered where continuous features are converted to discrete attribute values using the unsupervised 
K-Means clustering algorithm for vector quantization. The number of clusters (quantization 
levels) is empirically chosen for each feature. 

Compression-based measures represent a niche with valuable connections to the information-theoretical 
aspects of CBIR. Roman et al. (201 1) use the Normalized Compression Distance as a measure 
of similarity between two data files using the compression factor as an approximation to the 
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Kolmogorov complexity Cerra et al. (2011) instead proposes a Fast Compression Distance that 
can be efficiently computed between images and labels. 

6. Discussion 

EO mining evidently is a very lively research domain. Yet the issues to be overcome are 
still significant with respect to both basic research and operations. With particular reference to 
the main systems represented in the literature and whose attributes are reported in table 2, the 
issues of domain-specific (e.g. agriculture, disaster, urban etc) performance connected to basic 
architectures need to be taken into account together with aspects such as interoperability and 
scalability. The issue of domain-specific performances involves the specific optimizations that 
have been opted for in the engineering trade-oflFs typical of system design. Each one of the 
considered systems can also be seen as determined in its strenghts and limitations by the choices 
taken with the aim of a specific application domain. 

The GeoBrowse system allows experimentation in a high-level statistically oriented environment 
with rich data plotting and explorative data analysis capabilities. Examples in the published 
literature, though, mainly refer to LANDSAT data and to simple land cover classes. The relative 
richness of the operating environment might have proved a limiting factor as well as a competitive 
advantage for the system. Here the dilemma at the base of the design seems to have been 
flexibility versus ease of use. 

GeoIRIS is clearly optimized for metric resolution images of man-made objects. Security 
appears to be the central considered application. This well-specified focus has visibly impacted 
the system design, for instance in the way very effective yet perhaps scarcely flexible indices 
are built into the DBMS that serves as a back-end for the system. In the case of this system, 
performance and flexibility seem to have been the two competing considerations. 

A peculiarity of the I^KR system is its focus on the modeling of information sources by 

hybrid domain-specific ontologies used for data exploration and integration tasks. The tool 

has been exercised in the context of rapid mapping for disaster management in coastal areas: 

ontologies for Landsat and MODIS imagery based on the Anderson classification system have 
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Table 3: Cited references per EO mining system function. 
Systems and architectures 

King et al. (2007), Guo (2003), Tusk et al. (2003), Marcliisio et al. (1998), Shyu et al. (2006), Tobin et al. (2006), 
Schroder et al. (2000), Datcu et al. (2002), Datcu et al. (2003), Molinier et al. (2007b), 
Tobin et al. (2006), Marchisio et al. (1998), Datcu et al. (2003) 

Query specification 

Schroder et al. (2000), Datcu et al. (2003), Shyu et al. (2007), Klaric et al. (2006), Barb and Shyu (2007), 
Ben-Yitzhak et al. (2008), Marchisio et al. (1998), Marchisio and Cornelison (1999), Marchisio et al. (2000) 
Agouris et al. (1999) 

Labeling 

Datcu et al. (2003), Shyu et al. (2007), Aksoy (2007), Lienou and Campedel (2009), Lienou (2009), 

Lienou et al. (2010), Shah et al. (2004), Li and Narayanan (2004), Costache and Datcu (2007), Costache (2008), 

Costache et al. (2006a), Ferecatu et al. (2005), Ferecatu (2005), Ferecatu and Boujemaa (2007), Bordes and Prinet (2008), 

Aksoy et al. (2009), Aksoy and Cinbis (2010), Koperski and Marchisio (2000), Koperski et al. (2002b), Blanchart et al. (2011), 

Alber et al. (2001), Costache et al. (2006b), Barb and Shyu (2010a), Barb and Shyu (2010b), Ferecatu and Boujemaa (2007), 

Tula et al. (2009), Rital et al. (2008) 

Visualization 

Bratasanu et al. (201 1), Molinier et al. (2007b), Molinier et al. (2007a), Chen et al. (2008), Guo (2003), 
Guo et al. (2006), Guo and Gahegan (2006) Rivest et al. (2005) 



Query processing 



Durbha et al. (2008), Li et al. (2007), Vellaikal et al. (1995), Datcu et al. (2003), Aksoy et al. (2004), 

Durbha et al. (2005a), Durbha and King (2005), Durbha et al. (2005b), Durbha et al. (2007), Durbha et al. (2010), 

Ruan et al. (2006), Datta et al. (2006) 



Indexing 



Datcu et al. (2003), Shyu et al. (2006), Xiao and Liu (201 1), Tobin et al. (2006), Arya et al. (1998), 
Tobin et al. (2002), Shyu et al. (2007), Scott et al. (201 1) 



Segmentation 



Guo et al. (2009), Gueguen et al. (201 1), Shah et al. (2004), Datcu et al. (2003), Lienou et al. (2010), 
Durbha et al. (2007), Trias-Sanz et al. (2008) 



Primitive features 



Datcu et al. (2003), Aksoy and Cinbis (2010), Guo et al. (2009), Molinier et al. (2007b), Molinier et al. (2007a), 

Cerra et al. (201 1), Shyu et al. (2007), Schroder et al. (2000), Aksoy (2007), Tobin et al. (2006), 

Pietikainen et al. (2000), Yao and Chen (2003) Shah et al. (2005), Shah et al. (2007), Shah et al. (2006), 

Li and Bretschneider (2007), Dell'Acqua and Gamba (2001), Silva et al. (2005), Walessa and Datcu (2000), Aiyesha (2005), 

Tusk et al. (2003), Koperski et al. (2002a), Aksoy et al. (2002), Aksoy et al. (2005), Kalaycilar et al. (2008), 

Pesaresi and Benediktsson (2001), Zhang and Lu (2001), Matsakis (1998), Molinier et al. (2005) 



Multi-sensor 



Molch (2010), Molinier et al. (2007b), Molinier et al. (2007a), Molinier et al. (2008) 
Multi-temporal 



Heas and Datcu (2005), Gueguen and Datcu (2007), Gueguen and Datcu (2008), Mantovani et al. (2009), Natali et al. (201 1), 
Molinier et al. (2007a), Julea et al. (2006), Walter (2004) 



Discretization 



Cerra et al. (201 1), Datcu et al. (2003), Shyu et al. (2007), Maheshwary and Srivastava (2009), Aksoy (2006), 
Dong and Xiang-bin (2008), Bedawi and Kamel (201 1), Durbha et al. (2007), Aksoy (2007), Roman et al. (201 1), 
Mountrakis et al. (201 1) 



40 



been developed. Further ontologies for land cover characteristics have been conceptualized in 
the IGBP ontology and concepts in the hydrology domain have been formalized. A limitation in 
this approach might be related to the way the flexibility of the system is only obtainable by the 
proper modelling of the domain ontologies through experimentation by advanced users. 

KIM/KEO is probably the most flexible and interoperable among the considered systems. 
For instance, it allows users to easily include new feature extraction tools in the system. It has 
been applied in scenarios ranging from large archive management to flood mapping. Yet, focus 
on a specific application domain seems to be lacking. This in turn has made it probably difficult 
to focus on performance. KIM/KEO seems to be the result of choices that are the opposites of 
those at the basis of GeoIRIS. In this case, flexibility has been chosen on simplicity and sheer 
performance. 

The RBIR system is based on an innovative agent-based architecture. It is demonstrated on 
a large set of cover types ranging from forest to industrial. The primitive feature set considered 
is efficient in the considered application domain, yet the authors describe the need to extend it in 
order to support multi-temporal queries and to improve specificity via color information. Again, 
the performance might have been obtained at the partial expense of flexibility. 

The key application for the PicSOM system is related to the detection of buildings and the 
monitoring of construction and destruction events in couples of images. Experimentations on 
archives with larger data volumes and a broader spectrum of applications will be needed to 
evaluate the scalability of its performance to large scale operations conditions. 

The issue of scalability is of course a key one for all of the considered systems. KIM/KEO 
and GeoIRIS are arguably the tools that have been experimented with on the largest scales. Yet, 
the quantitative measurement of retrieval performance in the case of very large datasets requires 
the availability of ground truth information that can be extremely expensive to acquire. New 
strategies need to be devised in this regard (Quartulli et al., 2012). 

The picture that emerges shows that most of the available tools seem to be focusing on 
well-established archives from past missions rather than on currently active ones. The attributes 
in table 2 show how most of the tools deal with decametric data from either electro-optical 
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multi-spectral sensors. Limited support is available for more specialized sensor classes such as 
SAR and LIDAR. Multi-temporal analysis is present only in a very small handful of systems. 

While this is perfectly understandable based on historical reasons, it nevertheless might 
represent a significant limiting factor in the overall efficacy of novel missions with their increased 
data rates. It is very important that systems that are operational on Quickbird or SPOT data are 
effectively adapted to be used e.g. on the upcoming Sentinel data. 

The provision of systems such as those in Tobin et al. (2006); Costache and Datcu (2007) to 
understand the metric resolution data from sensors such as the WorldView, QuickBird, TerraS AR-X, 
COSMO Skymed, Sentinel systems is urgently needed, given their present and expected data 
production rates. 

Furthermore, for objectives related to rapid mapping and large-scale scene understanding in 
connection with large product archives, more effective tools for multi-sensor and multi-temporal 
EO mining and analysis are required. 

The specific usage patterns related to the management and exploitation of very large EO 
archives are to be taken into consideration as essential to the fulfillment of the promise of 
Earth observation for the next decades, and the issue of quantitatively evaluating the different 
approaches in order to measure their relative merits needs to be considered. 

A significant limitation of a number of approaches (e.g. the one in Mantovani et al. (2009) 
and Natali et al. (201 1) with respect to the one in Julea et al. (2006)) is the lack of prevision for 
incomplete knowledge of the data: the approaches are purely deterministic and does not foresee 
sound procedures for sensor noise and error treatment. 

Going back to the plot in figure 1 , we note that the temporal distribution of the considered 
citations shows that after a seminal phase that extends to the early 1990s, and after an initial 
implementation in 2000-2005, the second half of the last decade has been characterized by an 
explosion of activity. This growth seems now to be receding in a phase in which archive owners 
and operators are transitioning to operational systems for automatic data annotation connected 
directly to satellite payload ground segments. 
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7. Conclusions 

We have analyzed the state of the art of content-based retrieval in Earth observation remote 
sensing image archives with specific attention to complete systems showing promise for operational 
implementation. 

A generic system model has been presented and the functionality decomposition it defines 
has been used as a basis for the analysis of published approaches. 

The different paradigms at the basis of the main system families have been introduced. The 
approaches taken have been analyzed, focusing in particular on the phases after primitive feature 
extraction. The solutions envisaged for the issues related to feature simplification and synthesis, 
indexing, semantic labeling have been reviewed. The methodologies for query specification and 
execution have been analyzed. 

A large variety of approaches is present in the literature. Unfortunately, no general benchmarking 
tool and dataset are available for their evaluation. Quantitative benchmarking methods are needed 
to address the question of evaluating the efficacy of the engineering choices they embody. 

References 
References 

Abdi, H., Williams, L., 2010. Principal component analysis. Wiley Interdisciplinary Reviews: Computational Statistics 

2 (4), 433-459. 
Agouris, P., Carswell, J., Stefanidis, A., 1999. An environment for content-based image retrieval from large spatial 

databases. ISPRS Journal of Photogrammetry and Remote Sensing 54 (4), 263-272. 
Aiyesha, L. M., 2005. Index and retrieving of satellite imagery. Ph.D. thesis, Oakland University. 

URL http : //lielab- sees . sees . Oakland . edu/people/aiyesha/Papers/Thesis2005 . pdf 
Aksoy, S., 2006. Modeling of remote sensing image content using attributed relational graphs. In: Structural, Syntactic, 

and Statistical Pattern Recognition, pp. 475-483. 

URL http: //dx . doi . org/10 . 1007/11815921_52 
Aksoy, S., 2007. Spatial techniques for image classification. Image processing for remote sensing, 225. 

URL www.cs.bilkent.edu.tr/~saksoy/papers/crcpress05_spatial.pdf 
Aksoy, S., Cinbis, R., Jan. 2010. Image mining using directional spatial constraints. Geoscience and Remote Sensing 

Letters, IEEE 7(1), 33-37. 

43 



Aksoy, S., Koperski, K., Tusk, C, Marchisio, G. B., 2004. Interactive training of advanced classifiers for mining remote 

sensing image arcliives. In: Kim, W., Koliavi, R., Gehrke, J., DuMoucliel, W. (Eds.), Proceedings of tlie Tenth ACM 

SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, Washington, USA, August 

22-25, 2004. ACM, pp. 773-782. 

URL http: //doi . acm. org/10 . 1145/1014052 . 1016913 
Aksoy, S., Koperski, K., Tusk, C, Marchisio, G. B., May 2009. Land cover classification with multi-sensor fusion of 

partly missing data. Photogrammetric Engineering and Remote Sensing 75 (5), 577-593. 

URL http : //www . asprs . org/publicat ions/pers/2009j ournal /may /abstracts . html#577 
Aksoy, S., Koperski, K., Tusk, C, Marchisio, G. B., Tilton, J. C, 2005. Learning bayesian classifiers for scene 

classification with a visual grammar. IEEE T. Geoscience and Remote Sensing 43 (3), 581-589. 

URL http : //dx . doi . org/10 . 1109/TGRS . 2004 . 839547 
Aksoy, S., Marchisio, G. B., Koperski, K., Tusk, C, June 2002. Probabilistic retrieval with a visual grammar. In: 

Geoscience and Remote Sensing Symposium, 2002. IGARSS '02. 2002 IEEE International. Vol. 2. pp. 1041 - 1043. 
Alber, 1., Xiong, Z., Yeager, N., Farber, M., Pottenger, W. M., 2001. Fast retrieval of multi- and hyperspectral 

images using relevance feedback. In: Proc. IGARSS '01. IEEE 2001 International Geoscience and Remote Sensing 

Symposium, 09 Jul 2001 - 13 Jul 2001. Vol. 3. pp. 1149-1151. 
Arya, S., Mount, D., Netanyahu, N. S., Silverman, R., Wu, A. Y., 1998. An optimal algorithm for approximate nearest 

neighbor searching fixed dimensions. JACM: Journal of the ACM 45. 
Bahroun, S., Boujemaa, N., Belhadj, Z., 2010. Hierarchical visual thesaurus building for satellite image retrieval based 

on semantic region labelling. In: ICIP. IEEE, pp. 3193-3196. 

URL http : //dx . doi . org/10 . 1109/ICIP . 2010 . 5654151 
Baltsavias, E., 2004. Object extraction and revision by image analysis using existing geodata and knowledge: current 

status and steps towards operational systems. ISPRS Journal of Photogrammetry and Remote Sensing 58 (3), 129-151. 
Barb, A. S., Shyu, C.-R., 2007. User-specific semantics for modeling content-based geospatial knowledge. In: Proc. 

IGARSS 2007, Barcelona 23-28 July. IEEE, pp. 330-333. 

URL http : //dx . doi . org/10 . 1109/IGARSS . 2007 . 4422797 
Barb, A. S., Shyu, C.-R., 2010a. Visual information mining and ranking using graded relevance assessments in satellite 

image databases. In: Proc. IGARSS 2010. IEEE, pp. 3398-3401. 

URL http : //dx . doi . org/10 . 1109/IGARSS . 2010 . 5650173 
Barb, A. S., Shyu, C.-R., Jan. 2010b. Visual-semantic modeling in content-based geospatial information retrieval using 

associative mining techniques. Geoscience and Remote Sensing Letters, IEEE 7 (1), 38 -42. 
Bastian, M., Heymann, S., Jacomy, M., 2009. Gephi: An open source software for exploring and manipulating networks. 

URL http : //www . aaai . org/ocs/index . php/ICWSM/09/paper/view/154 
Bedawi, S. M., Kamel, M. S., 2011. Multiple classifier system for urban area's extraction from high resolution remote 

sensing imagery. In: Proceedings of the 8th international conference on Image analysis and recognition - Volume Part 

44 



II. ICIAR'll. Springer- Verlag, Berlin, Heidelberg, pp. 307-316. 

URL http : //dl . acm . org/citation . cf m?id=2027072 . 2027108 
Ben- Yitzhak, O., Golbandi, N., Har'El, N., Lempel, R., Neumann, A., Ofek-Koifman, S., Sheinwald, D., Shekita, E., 

Sznajder, B., Yogev, S., 2008. Beyond basic faceted search. In: Proceedings of the international conference on Web 

search and web data mining. WSDM '08. ACM, New York, NY, USA, pp. 33^4. 

URL http://doi.acm. org/10. 1145/1341531. 1341539 
Benz, U., Hofmann, R, Willhauck, G., Lingenfelder, I., Heynen, M., 2004. Multi-resolution, object-oriented fuzzy 

analysis of remote sensing data for gis-ready information. ISPRS Journal of Photogrammetry and Remote Sensing 

58 (3), 239-258. 
Bingham, A., McCleese, S., Stough, T., Deen, R. G., Hussey, K., Toole, N., oct. 2009. Earth science datacasting: 

Informed pull and information integration. Geoscience and Remote Sensing, IEEE Transactions on 47 (10), 3570 

-3580. 
Blanchart, P., Ferecatu, M., Datcu, M., July 2011. Mining large satellite image repositories using semi-supervised 

methods. In: Geoscience and Remote Sensing Symposium (IGARSS), 201 1 IEEE International, pp. 1585 -1588. 
Blaschke, T, 2010. Object based image analysis for remote sensing. ISPRS Journal of Photogrammetry and Remote 

Sensing 65(1), 2-16. 
Bordes, J. B., Prinet, V., 2008. Mixture distributions for weakly supervised classification in remote sensing images. In: 

BMVC. pp. xx-yy. 

URL http : //www . bmva . org/bmvc/2008/paper s/190 . pdf 
Boutemedjet, S., Ziou, D., Jan. 2008. A graphical model for context-aware visual content recommendation. Multimedia, 

IEEE Transactions on 10 (1), 52 -62. 
Bovolo, R, Bruzzone, L., Decembe 2007. Image information mining in time series: Algorithms and methods for 

prototyping. Tech. Rep. lIMTS-TN-ACS-0102 Issue 1.0, ESA. 
Bratasanu, D., Nedelcu, I., Datcu, M., 201 1 . Interactive spectral band discovery for exploratory visual analysis of satellite 

images. Selected Topics in Applied Earth Observations and Remote Sensing, IEEE Journal of PP (99), 1. 
Camara, G., Vinhas, L., Ferreira, K. R., Queiroz, G. R., Souza, R. C., Monteiro, A. M., Carvalho, M. T, Casanova, M. A., 

Freitas, U. M., 2008. TerraLib: An Open Source GIS Library for Large-Scale Environmental and Socio-Economic 

Applications. In: Open Source Approaches in Spatial Data Handling. Vol. 2 of Advances in Geographic Information 

Science. Springer Berlin Heidelberg, Berlin, Heidelberg, Ch. 12, pp. 247-270-270. 

URL http : //dx . doi . org/10 . 1007/978-3-540-74831- 1\_12 
Canty, M., 2007. Image analysis, classification and change detection in remote sensing: with algorithms for ENVI/IDL. 

CRC/Taylor & Francis. 

URL http : //books . google . es/books?id=mA20Yp8jU-4C 
Cerra, D., Bieniarz, J., Avbelj, J., Reinartz, P., Mueller, R., 201 1 . Compression-based unsupervised clustering of spectral 

signatures. In: Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), 2011 3rd 

45 



Workshop on. IEEE, pp. 1—4. 
Cerra, D., Mallet, A., Gueguen, L., Datcu, M., Jan. 2010. Algorithmic information theory-based analysis of earth 

observation images: An assessment. Geoscience and Remote Sensing Letters, IEEE 7 (1), 8 -12. 
Chaudhuri, S., Dayal, U., March 1997. An overview of data warehousing and olap technology. SIGMOD Rec. 26, 65-74. 

URL http : //doi . acm . org/10 . 1145/248603 . 248616 
Chen, J., MacEachren, A. M., Guo, D., 2008. Supporting the process of exploring and interpreting space-time 

multivariate patterns: The visual inquiry toolkit. Cartography and Geographic Information Science 35 (1), 33-50. 

URL http : //www . ncbi . nlm . nih . gov/pmc/art icles/PMC2786075/ 
Comon, R, 1996. Independent component analysis: a new concept. Signal Processing, Elsevier 36 (3), 287-314. 
Costache, M., 2008. Support vector machines and bayesian methods for category-based semantic learning: A search 

engine for image satellite database. Ph.D. thesis, TELECOM Paris Tech. 
Costache, M., Datcu, M., July 2007. Learning - unlearning for mining high resolution eo images. In: Geoscience and 

Remote Sensing Symposium, 2007. IGARSS 2007. IEEE International, pp. 4761 ^764. 
Costache, M., Lienou, M., Datcu, M., 2006a. On bayesian inference, maximum entropy and support vector machines 

methods. Aip Conference Proceedings 872 (1), 43-51. 

URL http: //link. aip.org/liiik/APCPCS/v872/il/p43/sl&Agg=doi 
Costache, M., Maitre, H., Datcu, M., 31 2006-aug. 4 2006b. Categorization based relevance feedback search engine for 

earth observation images repositories. In: Geoscience and Remote Sensing Symposium, 2006. IGARSS 2006. IEEE 

International Conference on. pp. 13 -16. 
Datcu, M., Daschiel, H., Pelizzari, A., QuartuUi, M., Galoppo, A., Colapicchioni, A., Pastori, M., Seidel, K., Marchetti, 

P. G., D'Elia, S., 2003. Information mining in remote sensing image archives: System concepts. IEEE Transactions 

on Geoscience and Remote Sensing 41 (12), 2923-2936. 
Datcu, M., D'Elia, S., King, R. L., Bruzzone, L., april 2007. Introduction to the special section on image information 

mining for earth observation data. Geoscience and Remote Sensing, IEEE Transactions on 45 (4), 795 -798. 
Datcu, M., King, R. L., D'Elia, S., Jan. 2010. Introduction to the special issue on image information mining: Pursuing 

automation of geospatial intelligence for environment and security. Geoscience and Remote Sensing Letters, IEEE 

7 (1), 3 -6. 
Datcu, M., Seidel, K., D'Elia, S., Marchetti, P. G., May 2002. Knowledge driven information mining in remote sensing 

image archives. ESA Bulletin, 26-33. 
Datta, R., Li, J., Parulekar, A., Wang, I. Z., 2006. Scalable remotely sensed image mining using supervised learning and 

content-based retrieval. Tech. Rep. CSE 06-019, Penn State University. 
De Carvalho, O., Meneses, P., 2000. Spectral correlation mapper (scm): An improvement on the spectral angle mapper 

(sam). In: Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) 2000 Workshop Proceedings, Pasadena. 
Dell'Acqua, R, Gamba, P., sep 2001. Query-by-shape in meteorological image archives using the point diffusion 

technique. Geoscience and Remote Sensing, IEEE Transactions on 39 (9), 1834 -1843. 

46 



Dong, W., Xiang-bin, W., 2008. Particle swarm intelligence classification algorithm for remote sensing images. 

In: PACIIA 2008, Volume 1, 2008 IEEE Pacific-Asia Workshop on Computational Intelligence and Industrial 

Application, 19-20 December 2008, Wuhan, China. IEEE Computer Society, pp. 566-570. 
Du, Y., Chang, C, Ren, H., Chang, C, Jensen, J., D'Amico, R, 2004. New hyperspectral discrimination measure for 

spectral characterization. Optical Engineering 43, 1777. 
Durbha, S., King, R., Shah, V., Younan, N., 2007. Image information mining for coastal disaster management. In: 

Geoscience and Remote Sensing Symposium, 2007. IGARSS 2007. IEEE International. IEEE, pp. 342-345. 
Durbha, S. S., King, R. L., 2005. Semantics-enabled framework for knowledge discovery from earth observation data 

archives. IEEE T. Geoscience and Remote Sensing 43 (II), 2563-2572. 

URL http : //dx . doi . org/10 . 1109/TGRS . 2005 . 847908 
Durbha, S. S., King, R. L., Gokaraju, B., Younan, N. H., July 2008. A proposal for the standardization of image 

information mining systems via ogc web services framework. In: Geoscience and Remote Sensing Symposium, 2008. 

IGARSS 2008. IEEE International. Vol. 3. pp. Ill -648 -III -651. 
Durbha, S. S., King, R. L., Younan, N. H., 2005a. Leveraging the semantic web technologies for knowledge discovery 

from earth observation data archives. In: ESA-EUSC 2005: Image Information Mining - Theory and Application to 

Earth Observation, Frascati, Italy. 
Durbha, S. S., King, R. L., Younan, N. H., July 2005b. Semantics driven framework for coastal zones. In: Geoscience 

and Remote Sensing Symposium, 2005. IGARSS '05. Proceedings. 2005 IEEE International. Vol. 8. pp. 5626 - 5629. 
Durbha, S. S., King, R. L., Younan, N. H., Jan. 2010. Wrapper-based feature subset selection for rapid image information 

mining. Geoscience and Remote Sensing Letters, IEEE 7 (1), 43 -Al . 
Ferecatu, M., July 2005. Image retrieval with active relevance feedback using both visual and keyword-based descriptors. 

Ph.D. thesis. University of Versailles Saint-Quentin-en-Yvelines. 
Ferecatu, M., Boujemaa, N., Apr. 2007. Interactive remote-sensing image retrieval using active relevance feedback. IEEE 

Trans. Geoscience and Remote Sensing 45 (4), 818-826. 

URL http : //dx . doi . org/10 . 1109/TGRS . 2007 . 892007 
Ferecatu, M., Crucianu, M., Boujemaa, N., Apr. 2005. Active SVM-based relevance feedback with hybrid visual and 

representation. 

URL http : //hal . lnria.fr/inria-00070448/en/ ; http : //hal . ccsd . cnrs . f r/docs/00/07/04/48/ 

PDF/RR- 5558.pdf 
Garramone, L., June 2009. The multimission national center of the Italian space agency. GSCB Workshop, 18th - I9th 

June 2009 ESA/ESRIN Frascati. 
Gautama, S., Bellens, R., deTre, G., Philips, W., Apr. 2007. Relevance criteria for spatial information retrieval using 

error-tolerant graph matching. IEEE Trans. Geoscience and Remote Sensing 45 (4), 810-817. 

URL http : //dx . doi . org/10 . 1109/TGRS . 2007 . 892006 
Giuliani, F, De Petris, M., Nico, G., 2010. Assessing scientific collaboration through coauthorship and content sharing. 

47 



Scientometrics 85 (1), 13-28. 
Gueguen, L., Datcu, M., Apr. 2007. Image time-series data mining based on the information-bottleneck principle. IEEE 

Trans. Geoscience and Remote Sensing 45 (4), 827-838. 

URL http : //dx . doi . org/10 . 1109/TGRS . 2006 . 890557 
Gueguen, L., Datcu, M., 2008. A similarity metric for retrieval of compressed objects: Application for mining satellite 

image time series. IEEE Trans. Knowl. Data Eng 20 (4), 562-575. 

URL http : //dx . doi . org/10 . 1109/TKDE . 2007 . 190718 
Gueguen, L., Pesaresi, M., Soille, R, 2011. An interactive image mining tool handling gigapixel images. In: IGARSS. 

IEEE, pp. 1581-1584. 

URL http : //leeexplore . ieee . org/xpl/mostRecentlssue . j sp?punumber=6034618 
Guns, R., Liu, Y., Mahbuba, D., 2011. Q-measures and betweenness centrality in a collaboration network: a case study 

of the field of informetrics. Scientometrics 87 (1), 133-147. 
Guo, D., 2003. Coordinating computational and visual approaches for interactive feature selection and multivariate 

clustering. Information Visualization 2 (4), 232-246. 

URL http://www.palgrave-journals.com/cgl-taf /DynaPage.taf?flle=/ivs/journal/v2/ii4/abs/ 

9500053a.html 
Guo, D., Chen, J., MacEachren, A. M., Liao, K., 2006. A visualization system for space-time and multivariate patterns 

(VIS-STAMP). IEEE Trans. Vis. Comput. Graph 12 (6), 1461-1474. 

URL http : //doi . ieeecomputersociety . org/10 . 1109/TVCG . 2006 . 84 
Guo, D., Gahegan, M., 2006. Spatial ordering and encoding for geographic data mining and visualization. J. Intell. Inf. 

Syst 27 (3), 243-266. 

URL http: //dx . doi . org/10 . 1007/sl0844-006-9952-8 
Guo, D., Xiong, H., Atluri, V, Adam, N. R., 2009. Object discovery in high-resolution remote sensing images: a semantic 

perspective. Knowl. Inf. Syst 19 (2), 211-233. 

URL http://dx.doi.org/10.1007/sl0115-008-0160-4 
Heas, P., Datcu, M., 2005. Modeling trajectory of dynamic clusters in image time-series for spatio-temporal reasoning. 

Geoscience and Remote Sensing, IEEE Transactions on 43 (7), 1635-1647. 
Hu, Y., 2005. Efficient, high-quality force-directed graph drawing. Mathematica Journal 10 (1), 37-71. 

URL http://www.mathematica-journal.com/issue/vlOil/contents/graph_draw/graph_draw.pdf 
Inglada, J., Christophe, E., July 2009. The orfeo toolbox remote sensing image processing software. In: Geoscience and 

Remote Sensing Symposium,2009 IEEE International,IGARSS 2009. Vol. 4. pp. IV-733 -IV-736. 
Ivanova, M., Nes, N., Goncalves, R., Kersten, M., 2007. Monetdb/sql meets skyserver: the challenges of a scientific 

database. In: Scientific and Statistical Database Management, 2007. SSBDM'07. 19th International Conference on. 

IEEE, pp. 13-13. 
Jaynes, E. T., 2003. Probability Theory - The Logic of Science - G. L. Bretthorst (Ed.). Cambridge University Press. 

48 



Julea, A., Meger, N., Trouve, E., 2006. On mining METEOSAT and ERS multitemporal images. In: 4tli Conf. on Image 

Information Mining for Security and Intelligence (ESA-EUSC 2006). 

URL http : //citeseerx . ist . psu . edu/viewdoc/suiimiary?dol=10 . 1 . 1 . 97 . 353 ;http : //earth . esa . 

int/rtd/Eveiits/ESA-EUSC_2006/0ral/Abl4_Meger.pdf 
Kalaycilar, R, Kale, A., Zamalieva, D., Aksoy, S., 2008. Mining of remote sensing image archives using spatial 

relationship histograms. In: IGARSS. IEEE, pp. 589-592. 

URL http : //dx . doi . org/10 . 1109/IGARSS . 2008 . 4779416 
Kao, D. T., Bergeron, R. D., Sparr, T. M., 1998. Efficient proximity search in multivariate data. In: Proceedings of 

the 10th International Conference on Scientific and Statistical Database Management. SSDBM '98. IEEE Computer 

Society, Washington, DC, USA, pp. 145-154. 

URL http : //dx . doi . org/10 . 1109/SSDM . 1998 . 688119 
Keogh, E., Lonardi, S., Ratanamahatana, C, 2004. Towards parameter-free data mining. In: Proceedings of the tenth 

ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp. 206-215. 
King, R. L., Durbha, S. S., Younan, N. H., 2007. Shared ontologies to facilitate semantic interoperability in earth 

observation archives. In: Proceedings of PV 2007 - Ensuring the Long-Term Preservation and Value Adding to 

Scientific and Technical Data. Wessling, Germany. 
King, R. L., Younan, N. H., Datcu, M., Nedelcu, I., 9 2011. Innovative data mining techniques in support of geoss: A 

workshop's findings. In: CORD Conference Proceedings, pp. 1^. 
Klaric, M., Scott, G. J., Shyu, C.-R., Davis, C. H., Palaniappan, K., 31 2006-aug. 4 2006. A framework for geospatial 

satellite imagery retrieval systems. In: Geoscience and Remote Sensing Symposium, 2006. IGARSS 2006. IEEE 

International Conference on. pp. 2457 -2460. 
Koperski, K., Marchisio, G. B., 2000. Multi-level indexing and GIS enhanced learning for satellite imageries. In: Simoft', 

S. J., Zaiane, O. R. (Eds.), Proceedings of the International Workshop on Multimedia Data Mining, MDM/KDD'2000, 

August 20th, 2000, Boston, MA, USA. University of Alberta, pp. 8-13. 
Koperski, K., Marchisio, G. B., Aksoy, S., Tusk, C, June 2002a. Visimine: interactive mining in image databases. In: 

Geoscience and Remote Sensing Symposium, 2002. IGARSS '02. 2002 IEEE International. Vol. 3. pp. 1810 - 1812 

vol.3. 
Koperski, K., Marchisio, G. B., Tusk, T, Aksoy, S., 2002b. Interactive models for semantic labeling of satellite images. 

In: SPIE Annual Meeting, Earth Observing Systems Session, Proceedings of SPIE, Seattle, WA, July 7-11. Vol. 4814. 

pp. 423-434. 
Korting, T. S., Fonseca, L. M., Escada, M. I. S., daSilva, R C, Silva, M. P S., 2008. GeoDMA - A novel system for 

spatial data mining. In: ICDM Workshops. IEEE Computer Society, pp. 975-978. 

URL http : //dx . doi . org/10 . 1109/ICDMW . 2008 . 22 
Kruse, P., Boardman, J., Huntington, J., 1999. Fifteen years of hyperspectral data: Northern grapevine mountains, nevada. 

In: Annual JPL Airborne Earth Science Workshop. Vol. 5. pp. 99-17. 

49 



Kruse, R, Lefkoff, A., Boardman, J., Heidebrecht, K., Shapiro, A., Barloon, P., Goetz, A., 1993. The spectral image 

processing system (sips)-interactive visualization and analysis of imaging spectrometer data. Remote sensing of 

environment 44 (2-3), 145-163. 
Li, J., Narayanan, R. M., 2004. Integrated spectral and spatial information mining in remote sensing imagery. IEEE T. 

Geoscience and Remote Sensing 42 (3), 673-685. 

URL http : //dx . doi . org/10 . 1109/TGRS . 2004 . 824221 
Li, J., Zlatanova, S., Fabbri, A., Zlatanova, S., 2007. Geomatics Solutions for Disaster Management. Lecture Notes in 

Geoinformation and Cartography. Springer. 

URL http : //books . google . ca/books?id=eoB6nTkhLqkC 
Li, Y., Bretschneider, T., april 2007. Semantic-sensitive satellite image retrieval. Geoscience and Remote Sensing, IEEE 

Transactions on 45 (4), 853 -860. 
Lienou, M., 2009. Apprentissage automatique de la semantique dans les images et representation en mots visuels des 

images satellitaires. Ph.D. thesis, Telecom ParisTech. 
Lienou, M., Campedel, M., 2009. Image semantic coding using OTB. In: IGARSS. IEEE, pp. 745-748. 

URL http : //dx . doi . org/10 . 1109/IGARSS . 2009 . 5417484 
Lienou, M., Maitre, H., Datcu, M., Jan. 2010. Semantic annotation of satellite images using latent dirichlet allocation. 

Geoscience and Remote Sensing Letters, IEEE 7 (1), 28 -32. 
Maheshwary, P., Srivastava, N., Aug. 2009. Prototype system for retrieval of remote sensing images based on color 

moment and gray level co-occurrence matrix. International Journal of Computer Science Issues, IJCSI. 

URL http : //cogprints . org/6702/1/3-20-23 . pdf ;http : //cogprints . org/6702/ 
Manning, C, Raghavan, P., Schiitze, H., 2009. Introduction to information retrieval. Cambridge University Press, 

Cambridge, England. 
Mantovani, S., Natali, S., Baraldi, A., 2009. Soil mapper multi-sensor and multi-temporal applications for semantic-based 

image information mining. In: Geospatial Visual Analytics - NATO Science for Peace and Security Series. Vol. C: 

Environmental Security Part 3. pp. 167-176. 
Marchisio, G. B., Cornelison, J., 1999. Content-based search and clustering of remote sensing imagery. In: Geoscience 

and Remote Sensing Symposium, 1999. IGARSS '99 Proceedings. IEEE 1999 International. Vol. 1. pp. 290 -292 

vol.1. 
Marchisio, G. B., Koperski, K., Sanella, M., 2000. Querying remote sensing and gis repositories with spatial association 

rules. In: Geoscience and Remote Sensing Symposium, 2000. Proceedings. IGARSS 2000. IEEE 2000 International. 

Vol. 7. pp. 3054 -3056 vol.7. 
Marchisio, G. B., Li, W. H., Sanella, M., Goldschneider, J., Jul 1998. Geobrowse: an integrated environment for satellite 

image retrieval and mining. In: Geoscience and Remote Sensing Symposium Proceedings, 1998. IGARSS '98. 1998 

IEEE International. Vol. 2. pp. 669 -673 vol.2. 
Matsakis, P., 1998. Relations spatiales structurelles et interpretation d'images. Ph.D. thesis, Institut de Recherche en 

50 



Informatique de Toulouse. 
Molch, K., Jan. 2010. Multisensor and muldtemporal fusion of vhr satellite imagery based on kim. Geoscience and 

Remote Sensing Letters, IEEE 7 (1), 48 -52. 
Molinier, M., Laaksonen, J., Ahola, J., Hame, T., 2005. Self-organizing map application for retrieval of man-made 

structures in remote sensing data. 

URL http://earth.esa.int/rtd/Events/ESA-EUSC_2005/Ar31_Molinier.pdf 
Molinier, M., Laaksonen, J., Hame, T., Apr. 2007a. Detecting man-made structures and changes in satellite imagery 

with a content-based information retrieval system built on self-organizing maps. IEEE Trans. Geoscience and Remote 

Sensing 45 (4), 861-874. 

URL http : //dx . doi . org/10 . 1109/TGRS . 2006 . 890580 
Molinier, M., Laaksonen, J., Vaatainen, S., Hame, T, 2007b. Self organising maps for change detection and monitoring 

of human activity in satellite imagery. In: Proc. of ESA-EUSC 2006 Image Information Mining for Security and 

InteUigence - WPP-274 (CD-ROM) Vol.March 2007. 

URL http : //earth . esa . int/rtd/Events/ESA-EUSC_2006/0ral/Ar08_Molinier .pdf 
Molinier, M., Viitaniemi, V., Koskela, M., Laaksonen, J., Rauste, Y., Hame, T, 2008. Improving content-based target 

and change detection in ALOS PALSAR images with efficient feature selection. In: Proceedings of ESA-EUSC 

Symposium 2008 - WPP-278 Vol.March 4-6. 

URL http : //earth . esa . int/rtd/Events/ESA-EUSC_2008/Papers/Ar27_Molinier . pdf 
Moon, T. K., 1996. The Expectation-Maximization algorithm. IEEE Signal Processing Magazine 11, 47-60. 
Mountrakis, G., Im, J., Ogole, C, 2011. Support vector machines in remote sensing: A review. ISPRS Journal of 

Photogrammetry and Remote Sensing 66 (3), 247-259. 
Munoz, I., Datcu, M., Jan. 2010. System design considerations for image information mining in large archives. 

Geoscience and Remote Sensing Letters, IEEE 7 (1), 13 -17. 
Natah, S., Beccati, A., D'Eha, S., VerateUi, M. G., Campalani, P., Folegani, M., Mantovani, S., July 2011. 

Multitemporal data management and exploitation infrastructure. In: Analysis of Multi-temporal Remote Sensing 

Images (Multi-Temp), 2011 6th International Workshop on the. pp. 217 -220. 
Neteler, M., Raghavan, V., et al., 2006. Advances in free software geographic information systems. Journal of Informatics 

3(2). 
Padwick, C, Pacifici, R, Marchisio, G. B., Logbotham, N., 2011. Fusion of morphological and texture features for 

high resolution image change detection. In: ASPRS 2011 Annual Conference Milwaukee, Wisconsin May 1-5. pp. 

398^07. 
Pahl, M., 2008. Arquitetura de um sistema baseado em conhecimento para a interpreta^ao de dados de sensoriamento 

remoto de miiltiplos sensores. Ph.D. thesis, INPE Sao Jose dos Campos. 
Peijun, D., Yunhao, C, Hong, T, Tao, P., July 2005. Study on content-based remote sensing image retrieval. In: 

Geoscience and Remote Sensing Symposium, 2005. IGARSS '05. Proceedings. 2005 IEEE International. Vol. 2. 

51 



p. 4 pp. 
Pesaresi, M., Benediktsson, J., feb 2001 . A new approach for the morphological segmentation of high-resolution satellite 

imagery. Geoscience and Remote Sensing, IEEE Transactions on 39 (2), 309 -320. 
Pietikainen, M., Ojala, T., Xu, Z. L., Jan. 2000. Rotation-invariant texture classification using feature distributions. 

Pattern Recognition 33 (1), 43-52. 

URL http : //www. sclencedirect . coiii/science/article/B6V14-40CKlNW-4/2/ 

1 cd5 Id6725d5da7a966a8f 64793f a3d9 
QuartuUi, M., Zorrilla, M., Olaizola, I., Foreseen for October, 2012 2012. On the image content of the esa eusc jrc 

workshop on image infomation mining, submitted for publication on the Proceedings of the 2012 ESA EUSC JRC 

Workshop on Image Information Mining to be held in Oberpfaffenhofen, Germany. 
Research Systems, I., 2003. ENVI user's guide. Research Systems. 

URL http : //books . google . es/books?id=WwnuAAAAMAA J 
Rissanen, J., 1985. Minimum-Description-Length principle. In: Encyclopedia of Statistical Sciences. Vol. 5. Wiley, New 

York, pp. 523-527. 
Rital, S., Costache, M., Campedel, M., December 2008. Plato for information mining in satellite imagery. In: Semantic 

and Digital Media Technologies (SAMT), Koblenz, Germany. 
Rivest, S., Bedard, Y., Proulx, M., Nadeau, M., Hubert, P., Pastor, J., 2005. Solap technology: Merging business 

intelligence with geospatial technology for interactive spatio-temporal exploration and analysis of data. ISPRS journal 

of photogrammetry and remote sensing 60 (1), 17-33. 
Roman, A. G., Veganzones, M. A., Gratia, M., Datcu, M., 201 1. A novel data compression technique for remote sensing 

data mining. In: Proceedings of ESA JRC EUSC 2011 Image Information Mining Conference. 
Ruan, N., Huang, N., Hong, W., 31 2006-aug. 4 2006. Semantic-based image retrieval in remote sensing archive: 

An ontology approach. In: Geoscience and Remote Sensing Symposium, 2006. IGARSS 2006. IEEE International 

Conference on. pp. 2903 -2906. 
Samet, H., Soifer, A., 1994a. A legend-driven geographic symbol recognition system, pp. B:350-355. 
Samet, H., Soifer, A., 1994b. Magellan: Map acquisition of geographic labels by legend analysis. 
Samet, H., Sofi'er, A., 1996. Marco: Map retrieval by content. IEEE TRANSACTION ON PATTERN ANALYSIS AND 

MACHINE INTELLIGENCE 18, 783-798. 
Schroder, M., Rehrauer, H., Seidel, K., Datcu, M., 2000. Interactive learning and probabilistic retrieval in remote sensing 

image archives. IEEE Transactions on Geoscience and Remote Sensing 38, 2288-2298. 
Scott, G. J., Klaric, M., Davis, C. H., Shyu, C.-R., 2011. Entropy-balanced bitmap tree for shape-based object retrieval 

from large-scale satellite imagery databases. IEEE T. Geoscience and Remote Sensing 49 (5), 1603-1616. 

URL http : //dx . doi . org/10 . 1109/TGRS . 2010 . 2088404 
Scott, G. J., Klaric, M., Shyu, C.-R., 2005. Modeling multi-object spatial relationships for satellite image database 

indexing and retrieval. In: CIVR. pp. 247-256. 

52 



URL http : //dx . doi . org/10 . 1007/11526346_28 
Shah, V. P., Durbha, S. S., Younan, N. H., King, R. L., 2004. A wavelet-based approach for knowledge mining in earth 

observation data archives. In: Proc. ESA-EUSC 2005: Image Information Mining - Tehory and Application to Earth 

Observation, Frascati, Italy. 
Shah, V. P., Durbha, S. S., Younan, N. H., King, R. L., 31 2006-aug. 4 2006. Coalescing ica and wavelets coefficients for 

image information mining in earth observation data archives. In: Geoscience and Remote Sensing Symposium, 2006. 

IGARSS 2006. IEEE International Conference on. pp. 9 -12. 
Shah, V. P., Younan, N. H., Durbha, S. S., King, R. L., July 2005. Wavelet features for information mining in remote 

sensing archives. In: Geoscience and Remote Sensing Symposium, 2005. IGARSS '05. Proceedings. 2005 IEEE 

International. Vol. 8. pp. 5630 - 5633. 
Shah, V. P., Younan, N. H., Durbha, S. S., King, R. L., Apr. 2007. A systematic approach to wavelet-decomposition-level 

selection for image information mining from geospatial data archives. IEEE Trans. Geoscience and Remote Sensing 

45 (4), 875-878. 

URL http : //dx . doi . org/10 . 1109/TGRS . 2007 . 892005 
Sheikholeslami, G., Chang, W., Zhang, A., sep/oct 2002. Semquery: semantic clustering and querying on heterogeneous 

features for visual data. Knowledge and Data Engineering, IEEE Transactions on 14 (5), 988 - 1002. 
Shyu, C.-R., Klaric, M., Scott, G. J., Barb, A. S., Davis, C. H., Palaniappan, K., april 2007. Geoiris: Geospatial 

information retrieval and indexing system - content mining, semantics modeling, and complex queries. Geoscience 

and Remote Sensing, IEEE Transactions on 45 (4), 839 -852. 
Shyu, C.-R., Klaric, M., Scott, G. J., Mahamaneerat, W. K., 31 2006-aug. 4 2006. Knowledge discovery by mining 

association rules and temporal-spatial information from large-scale geospatial image databases. In: Geoscience and 

Remote Sensing Symposium, 2006. IGARSS 2006. IEEE International Conference on. pp. 17 -20. 
Silva, M. P. S., Camara, G., Souza, R. C, Valeriano, D., Escada, M. I. S., nov. 2005. Mining patterns of change in remote 

sensing image databases. In: Data Mining, Fifth IEEE International Conference on. p. 8 pp. 
Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., Jain, R., 2000. Content-based image retrieval at the end of the 

early years. IEEE Trans. Pattern Anal. Mach. Intell. 22 (12), 1349-1380. 
Soccorsi, M., Quartulli, M., Datcu, M., 2006. Space- Variant model fitting and selection for image information extraction. 

In: AIP Conf. Proc. Bayesian Inference and Maximum Entropy Methods In Science and Engineering - November 29. 

Vol. 872. pp. 358-366. 
Softer, A., Samet, H., 1998. Magellan: Map acquisition of geographic labels by legend analysis 1 (2), 319-330. 
Tobin, K. W., Bhaduri, B. L., Bright, E. A., Cheriyadat, A. M., Kamowski, T. P, Palathingal, P J., Potok, T E., 

Price, J. R., May 2006. Automated feature generation in large-scale geospatial libraries for content-based indexing. 

Photogrammetric Engineering and Remote Sensing 72 (5), 531-540. 

URL http : //www . asprs . org/publicat ions/pers/2006j ournal /may /abstracts . html#531 
Tobin, K. W., Karnowski, T. P., Arrowood, L. F, Ferrell, R. K., Goddard, J. S., Lakhani, F, Jul. 2002. Content-based 

53 



image retrieval for semiconductor process characterization. EURASIP Journal on Applied Signal Processing 2002 (7), 

704-713. 
Trias-Sanz, R., Stamon, G., Louchet, J., 2008. Using colour, texture, and hierarchial segmentation for high-resolution 

remote sensing. ISPRS Journal of Photogrammetry and remote sensing 63 (2), 156-168. 
Tuia, D., Ratle, R, Pacifici, R, Kanevski, M., Emery, W., 2009. Active learning methods for remote sensing image 

classification. Geoscience and Remote Sensing, IEEE Transactions on 47 (7), 2218-2232. 
Tusk, C., Koperski, K., Aksoy, S., Marchisio, G. B., July 2003. Automated feature selection through relevance feedback. 

In: Geoscience and Remote Sensing Symposium, 2003. IGARSS '03. Proceedings. 2003 IEEE International. Vol. 6. 

pp. 3691 - 3693 vol.6. 
Vapnik, V., sep 1999. An overview of statistical learning theory. Neural Networks, IEEE Transactions on 10 (5), 988 

-999. 
Vellaikal, A., Kuo, C. C, Dao, S., 1995. Content-based retrieval of remote sensed images using vector quantization. In: 

Proceedings of SPIE Visual Information Processing 1995, Orlando USA. Vol. 4. 

URL http://citeseerx. ist .psu.edu/viewdoc/summary?doi=10. 1 . 1.55. 2391; http://biron.usc. 

edu/~vellaika/papers/vip95 .ps 
Walessa, M., Datcu, M., 2000. Model-based despeckling and information extraction from SAR images. Geoscience and 

Remote Sensing, IEEE Transactions on 38 (5), 2258-2269. 
Walker, A. R., Pham, B. L., Moody, M. P., 2005. Spatial bayesian learning algorithms for geographic information 

retrieval. In: 13th annual ACM international workshop on Geographic information systems. ACM Press, Bremen, 

Germany, pp. 105-114. 

URL http : //sprints . qut . edu . au/3637/ 
Walter, V, 2004. Object-based classification of remote sensing data for change detection. ISPRS Journal of 

Photogrammetry and Remote Sensing 58 (3), 225-238. 
Widmann, N., Baumann, P., 1998. Efficient execution of operations in a dbms for multidimensional arrays. In: 

Proceedings of the 10th International Conference on Scientific and Statistical Database Management. SSDBM '98. 

IEEE Computer Society, Washington, DC, USA, pp. 155-165. 

URL http://dx.doi.org/10.1109/SSDM. 1998.688120 
Xiao, Z., Liu, Y., June 2011. Remote sensing image database based on nosql database. In: Geoinformatics, 2011 19th 

International Conference on. pp. 1 -5. 
Yao, C. H., Chen, S. Y., Apr. 2003. Retrieval of translated, rotated and scaled color textures. Pattern Recognition 36 (4), 

913-929. 

URL http : //www. sciencedirect . coiii/science/article/B6V14-47FlHVN-4/2/ 

2849e74eafad851al4748f7add030e98 
Zhang, D., Lu, G., aug. 2001. Content-based shape retrieval using different shape descriptors: a comparative study. In: 

Multimedia and Expo, 2001. ICME 2001. IEEE International Conference on. pp. 1139 - 1142. 

54 



